nvidia-cdi-hook fails when running a container with --userns nomap

[lahwaacz]

Running a container with --gpus all and either --userns host or --userns keep-id works, but --userns nomap fails. Note that this was tested on Arch Linux with nvidia-container-toolkit 1.16.1.

First with podman 5.2.0 and crun 1.16:

$ podman run --rm -it --gpus all --runtime /usr/bin/crun --userns host quay.io/archlinux/archlinux:base
[root@11d41940527d /]#
exit
$ podman run --rm -it --gpus all --runtime /usr/bin/crun --userns keep-id quay.io/archlinux/archlinux:base
[lahwaacz@17ff7ae4f275 /]$
exit
$ podman run --rm -it --gpus all --runtime /usr/bin/crun --userns nomap quay.io/archlinux/archlinux:base
Error: OCI runtime error: /usr/bin/crun: {"msg":"error executing hook `/usr/bin/nvidia-cdi-hook` (exit code: 1)","level":"error","time":"2024-08-13T07:48:19.461304Z"}

With podman 5.2.0 and runc 1.1.13:

$ podman run --rm -it --gpus all --runtime /usr/bin/runc --userns host quay.io/archlinux/archlinux:base
[root@6b07c771066e /]#
exit
$ podman run --rm -it --gpus all --runtime /usr/bin/runc --userns keep-id quay.io/archlinux/archlinux:base
[klinkovsky@6083384b42e7 /]$
exit
$ podman run --rm -it --gpus all --runtime /usr/bin/runc --userns nomap quay.io/archlinux/archlinux:base
Error: /usr/bin/runc: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2024-08-13T09:49:32+02:00" level=error msg="failed to determined container root: failed to open OCI spec file: open /home/lahwaacz/.local/share/containers/storage/overlay-containers/981df8bef771d05f1e2f2a907cf6a0d0698ab2f24dedb34201ef14d91c6959cc/userdata/config.json: permission denied": OCI permission denied

The failed to determined container root: failed to open OCI spec file message (notable for its grammar error) comes from update-ldcache.go:

https://github.com/NVIDIA/nvidia-container-toolkit/blob/4f0de9f1ef07abd6c1b3dd54995b275fb74d14dc/cmd/nvidia-cdi-hook/update-ldcache/update-ldcache.go#L103-L106

(Unrelated rant at crun and runc: neither of their output is complete, but they complement each other - crun shows which hook failed and runc shows the actual error message...)

[lahwaacz]

Using --userns auto leads to the same error as --userns nomap.

[elezar]

The hook is trying to open the OCI runtime specification for the container so as to determine the container root. I assume that with the --userns auto|nomap option, the hook is being run as a user that does not have read access to the path shown in the runc output. Since nomap.

The container root is required so as to properly update the ldcache in the container -- or skip the update of the ldcache if it is not present.