k3s: GPU Passthrough for Nvidia / AMD / Intel

[superherointj]

This issue is for tracking GPU pass through in K3s.

K3s supports GPU pass through but not in NixOS K3s (last time I tried).

I don't know if it is a solved issue. I think it is not, but:

• there are notes on how to do GPU passthrough here:
  • https://wiki.nixos.org/wiki/K3s (Let me know if it works for you. I will test too. And leave my notes here.)

Last time I tried, I had issues with nix paths, ldcache in Nvidia driver. I got lost in the process. I will keep updating this issue until: GPU pass through is:

• supported by NixOS K3s.
• properly documented
• integrated to NixOS K3s module. (Ideally GPU pass through should be a toggle.)

Other references:

• https://discourse.nixos.org/t/using-nvidia-container-runtime-with-containerd-on-nixos/27865/30
• https://github.com/NixOS/nixpkgs/pull/279235

[SomeoneSerge]

Related: https://github.com/NixOS/nixpkgs/pull/278969 https://github.com/NixOS/nixpkgs/pull/284507

[OlfillasOdikno]

I was unable to get the official NVIDIA device plugin to work, since all the heavy lifting is already done in containerd and generating cdi json, I created a device plugin that uses the cdi json and instructs kubernetes to inject the device. Tested it on a NVIDIA 3060. https://github.com/OlfillasOdikno/generic-cdi-plugin

[ahirner]

@OlfillasOdikno thanks, I use your plugin as well for now. I had problems that some containers didn't see libnvml.so.1, nor the generated CDIs.

GPU pass through is:

Question regarding scope: does this issue inlcude shared GPU use? I'm not sure how involved it is.