search

nebius / soperator

Run Slurm in Kubernetes
Apache License 2.0
101 stars 9 forks source link

AMD GPUs support #81

Open ybannay opened 1 month ago

ybannay commented 1 month ago

Hi guys,

I played with the operator and it worked well for NVIDIA GPUs, but not for AMD. Do you have in mind to add support for AMD GPUs or if we can contribute to the project adding the support.

Thanks

rdjjke commented 1 month ago

Hello.

Thank you for the first issue in this repo!

We’re currently focused on improving the stability of Soperator (tests and the like), so we had no plans to support other GPUs shortly.

But if you implement it yourself, we will help you.

However, the task looks difficult. For NVIDIA GPUs, we use nvidia-container-toolkit for propagating device drivers and some low-level libraries from the “Host” to the “Jail” environment. I'm not sure if there is anything like that for AMD. Anyway, the main difficulties will be with container images, not with the operator code.

rdjjke commented 1 month ago

BTW, you said that you played with the operator, and in what environment (cloud?) if it's not a secret? We haven't tested Soperator anywhere outside of Nebius Сloud, so I'm wondering where else we can assume it works.