Kubernetes operator

[davidspek]

Not really an issue, but I’ve had this idea to use a Kubernetes operator to spin up pods for transcoding for a while now. Not sure if this project would be a good starting point for this but thought I’d share the idea. It might also be similar to https://github.com/ccremer/clustercode. This would probably require some more integration on the Jellyfin side as well so when a new transcode is started it makes some sort of call which would then trigger the creation of a transcoding pod.

[joshuaboniface]

Not a bad idea at all and I believe it's been asked about in the past, if not for Kube specifically then for WoL (e.g. #21, #29).

Probably the easiest way to integrate this (since I don't see something that specific falling under Jellyfin's purvue) would be to write a wrapper script that triggers pod creation then launches rffmpeg using the right alias. So something like...

$ cat /usr/local/bin/ffmpeg-pod
#!/usr/bin/env bash
do_trigger_pod_creation
/usr/local/bin/ffmpeg "$@"

I imagine that would work; only potential caveat being option quoting but I think "$@" will handle that right.

[aleksasiriski]

Not really an issue, but I’ve had this idea to use a Kubernetes operator to spin up pods for transcoding for a while now. Not sure if this project would be a good starting point for this but thought I’d share the idea. It might also be similar to https://github.com/ccremer/clustercode. This would probably require some more integration on the Jellyfin side as well so when a new transcode is started it makes some sort of call which would then trigger the creation of a transcoding pod.

You could just use rffmpeg-worker docker image as a deployment and add HPA, or use it as a daemonset, as well as cluster autoscaler.

[schklom]

This sounds similar to https://github.com/HaveAGitGat/Tdarr

[aleksasiriski]

This sounds similar to https://github.com/HaveAGitGat/Tdarr

Tdarr, like Unmanic, is used to convert all of your media to a single format. It's independent of Jellyfin. Also Tdarr is now closed sourced

[schklom]

I had not noticed they were closed-source. Thanks for letting me know :)

[lknite]

For context, I'm just starting to look at jellyfin having used kube-plex in the past (which spins up pods for transcoding, but no longer works due to changes in plex).

This git repo seems like it might have the answer to creating something similar with jellyfin.

I'm hearing suggestions from total pros, however, the words are pretty greek to me. So implementing these suggestions might be quite a challenge at this point.

I could probably write an operator but don't really want to reinvent the wheel. It almost sounds like a helm chart, along with maybe a more customized jellyfin image might get us there? Does adding rffmpeg require a custom image?

Any progress made by anyone on horizontal scaling? I'm assuming Jellyfin itself isn't made to scale in that way ...

[lknite]

Maybe we could use NATS to submit a "transcoding job message", which would then be consumed by a transcoding-controller, the controller could spin up a pod to do the actual transcoding & manage its lifecycle.

Submitting a task using a message queue results in something that can scale, on the sending side multiple replicas could potentially send the request, and on the receiving side the 'message subscriber' could be any number of replicas.

The existing helm chart could be enhanced to include an option to deploy NATS & also to enable submitting transcoding jobs to a message queue instead of whatever the default is.

I could potentially help with all the above, unless it could possibly be added to the roadmap (@joshuaboniface ) but I don't know what would run in the pod to perform the actual transcoding or what would need to talk to what.

A transcoder-controller could potentially maintain a min/max number of pods ready to transcode, or maybe that could be done automatically using the typical kubernetes pod autoscaler. In anycase, sending the transcoding request message to a queue would open the door to folks who might interested to put something together.

[lknite]

Maybe a scaling architecture like this?

• 1 or more replicas of the jellyfin binary (frontend)
• a loadbalancer can round robin to any jellyfin instance, and it will work
• a backend database that can scale such as: cockroachdb, so there are no bottlenecks in the future
• when its time to transcode, a message is sent to a message queue such as NATS
• 1 or more replicas of transcoder-controller is subscribed to the message queue, and spins up a transcoding pod as needed, or otherwise maintains a pool of pods ready to transcode

[loadbalancer]
  -> [ jellyfin ] [ jellyfin ] [ jellyfin ]
    -> [ cockroachdb ]
    -> NATS Message Queue 'Transcode Job'

[NATS Message Queue]
  -> [ transcode-controller ] [ transcode-controller ]
    -> Spin up transcoding_pod as needed

[ transcode_pod ] [ transcode_pod ] [ transcode_pod ] [ transcode_pod ] [ transcode_pod ] [ transcode_pod ]
[ transcode_pod ] [ transcode_pod ] [ transcode_pod ] [ transcode_pod ] [ transcode_pod ] [ transcode_pod ]

Should be easy to run as standalone like in current installs, and just need a way to 'submit transcode to message queue' that can be enabled to work with the scaling options.

What would a pod look like that just does the transcoding step?

[aleksasiriski]

Hello there, took a quick look at your messages, you can take a look at my repos for some attempts on doing this (ffmpeg-of & jellyfin kubernetes helm chart). Might give you some hints, although I'm not maintaining any of it anymore since I had other priorities.

Hopefully some day someone rewrites Jellyfin backend API in a microserviced architecture so that this scaling is taken care of :)

[lknite]

@aleksasiriski it looks like maybe you got it all working? did you not? did you run into issues?

[aleksasiriski]

I had it working a while back, using pure K8s yaml files, but then someone from the community converted that into a Helm chart which I never tested personally. I mainly focused on developing ffmpegof at the time, which is a rewrite of this repo using Go with Jellyfin docker images created with it. Since then. I've moved on to different things.

[psych0d0g]

take a look at my shot at it, i dont have HPA running for it yet, but my project https://github.com/CrystalNET-org/grpc-ffmpeg might be perfectly suited for doing that :) i have a prometheus metrics endpoint implemented exposing the current running transcode sessions, one might be able to add a HPA configuration based on that to scale the deployment up when the current paralell transcoding sessions exceed a specific number on a node (altough im just running it as a static daemonset currently) i have the accompanying jellyfin container with the client part integrated aswell (atough that repo lacks documentation right now): https://github.com/CrystalNET-org/jellyfin if you wanna have a chat to brainstorm or something, im hanging around in joshua's matrix chatroom, or on discord (check the crystalnet github organisation for the link to Discord)