search

pabloromeo / clusterplex

ClusterPlex is an extended version of Plex, which supports distributed Workers across a cluster to handle transcoding requests.
MIT License
469 stars 36 forks source link

kubernetes / i915 gpu, no hardware acceleration option #195

Closed cranky-coder closed 1 year ago

cranky-coder commented 1 year ago

I have my cluster configured to run the workers on nodes that have the i915 intel gpu enabled. I have it configured in the resource limits and they are being assigned to the right nodes. But when I watch the gpu usage on the nodes, there is 0 gpu work happening but the cpu's are getting hit hard. In the settings for the clusterplex instance i do not see the option to enable hardware transcoding.

image

Any suggestions?

pabloromeo commented 1 year ago

Hi! So at the moment my cluster isn't using the iGPU so I don't have a working example. Hopefully I'll find time to set that up soon.

But I believe you need to have Intel's drivers running on all Nodes: https://github.com/intel/intel-device-plugins-for-kubernetes/

I expect you also need to adapt the clusterplex worker deployments to do the resource request for i915.

From the Clusterplex side of things, it seems it should only require you to specify the FFMPEG_HWACCEL environment setting with the appropriate value on the Worker deployments.

Similar to what's described here: https://www.reddit.com/r/selfhosted/comments/vgk4e9/plex_on_kubernetes_with_hardware_decoding_victory/

cranky-coder commented 1 year ago

Checking that last one!! I have the device drivers in the nodes, and updated the deployment, but that ENV I didn't change!! that might be it!

pabloromeo commented 1 year ago

Nice! Yeah, I'm not sure what the proper value for FFMPEG_HWACCEL would be though, for i915, not sure if it's QSV or VAAPI or something else. Once you get it working, if you want to share the deployment yaml and info on running the driver I can add it to the documentation, as an example with HW transcoding.

meyayl commented 1 year ago

This is what you get on i915 with FFMPEG_HWACCEL: QSV

plex_worker  | Unrecognized hwaccel: QSV.
plex_worker  | Supported hwaccels: cuda vaapi drm opencl
cranky-coder commented 1 year ago

I used vaapi. Works so far. Gonna do a PR hopefully this weekend with my full yaml.

pabloromeo commented 1 year ago

Awesome!! Glad to hear it's working. Hopefully I'll be able to replicate the setup soon. Since I run it on k3s in VMs in proxmox, gotta first do a bit of research on passthrough for the iGPU to get it to work.

pabloromeo commented 1 year ago

Hi @cranky-coder, if you ultimately got i915 to work on your k8s cluster, could you share your config? I believe others, such as in this issue https://github.com/pabloromeo/clusterplex/issues/223 may benefit from a working example.

evanrich commented 6 months ago

Curious about this too. I have the i915 drivers installed, intel pods all show 915 available, but the workers can't find the gpu. I tried setting the FFMPEG_HWACCEL=vaapi but it doesn't seem to find them. running talos version of kubernetes if that helps. I've seen that you may need to set a pod secuirity policy to allow privledged access, but i dont think the chart supports that.