elezar
commented
5 months ago The error:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices nvidia.com/gpu=all: unknown.Indidates that rootless docker cannot find the CDI specifications that were generated. As far as I am aware, rootless docker modifies the path used for /etc (and other paths) and this is what could be causing issues here for the runtime.
Since you're using a docker version that supports CDI (as an opt-in feature, I believe). Could you try the native CDI injection here.
Running:
nvidia-ctk runtime configure --runtime=docker --cdi.enabledand restarting the docker daemon should enable this feature. (Note that the command may need to be adjusted for rootless mode to specify the config file path explicitly as per https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#rootless-mode).
Then with the CDI feature enabled in docker you should be able to run:
$ docker run --rm -ti --device=nvidia.com/gpu=all ubuntu nvidia-smi -Land have the devices injected without using the nvidia runtime.
LukasIAO
klueska
Milor123
kirovtome
Hello everyone,
we have recently set up a rootless docker instance alongside our existing docker on one of our servers, but ran into issues mounting host GPUs into the rootless containers. A workaround was presented in issue #85 (toggling no-cgroups to switch between rootful and rootless) with a mention of a better solution in the form of Nvidia CDI coming as an experimental feature in Docker 25.
After updating to the newest Docker releases and setting up CDI, our regular Docker instance behaved as we expected based on the documentation, but the rootless instance still runs into issues.
Setup to reproduce:
config.toml (click to expand)
``` #accept-nvidia-visible-devices-as-volume-mounts = false #accept-nvidia-visible-devices-envvar-when-unprivileged = true disable-require = false #swarm-resource = "DOCKER_RESOURCE_GPU" [nvidia-container-cli] #debug = "/var/log/nvidia-container-toolkit.log" environment = [] #ldcache = "/etc/ld.so.cache" ldconfig = "@/sbin/ldconfig.real" load-kmods = true #no-cgroups = true #no-cgroups = false #path = "/usr/bin/nvidia-container-cli" #root = "/run/nvidia/driver" #user = "root:video" [nvidia-container-runtime] #debug = "/var/log/nvidia-container-runtime.log" log-level = "info" mode = "auto" runtimes = ["docker-runc", "runc"] [nvidia-container-runtime.modes] [nvidia-container-runtime.modes.csv] mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d" ```nvidia.yaml (click to expand)
The issue: When
no-cgroups = falseCDI injection works fine for the regular Docker instance:but produces the following errors for the rootless version:
Running
docker run --rm --gpus all ubuntu nvidia-smiresults in the same error as without OCI. This seems to be consistent across all variations listed on the Specialized Configurations for Docker page:Interestingly, setting
no-cgroups = truedisables the regular use of GPUs with rootful Docker:but still allows for CDI injections:
With control groups disabled, the rootless daemon is able to use exposed GPUs as outlined in the Docker docs:
TLDR Disabling c-groups allows the rootless containers to use exposed GPUs using the regular docker run --gpus flag. This in turn disables the rootful container's GPU access. Leaving control groups enabled reverses the effect, as outlined in #85 .
Disabling c-groups and using Nvidia CDI, the rootful Docker can still use GPU injection, even though regular GPU access is barred, while the rootless container uses the exposed GPUs. CDI injection for rootless fails in both cases, however.
This seems like a definite improvement, but I'm not sure it's intended behavior. The CDI injection failing with rootless regardless of control group setting leads me to believe this is unintended, unless rootless is not yet supported by Nvidia CDI.
Any insights would be greatly appreciated!