secondspass
commented
3 years ago I don't know if this is the right place to ask, and I can open a separate issue if needed.
I'm testing rootless Podman v3.0 with crun v0.17 on our Summit test systems at Oak Ridge (IBM Power 9 with Nvidia Tesla V100 GPUs, RHEL 8.2). We have a restriction that we can't setup and maintain the subuid/subgid mappings for each of our users in the /etc/sub[uid|gid] files. That would be a giant administrative overhead since that mapping would have to be maintained on every node. Currently pulling or building cuda containers works just fine. But when trying to run it.
% podman run --rm --security-opt=label=disable --hooks-dir=/usr/share/containers/oci/hooks.d/ oci-archive:/ccs/home/subil/subil-containers-oci/simplecuda nvidia-smi
Getting image source signatures
Copying blob 5ef3c0b978d0 done
Copying blob d23be3dac067 done
Copying blob 786d8ed1601c done
Copying blob 6e99435589e0 done
Copying blob 93d25f6f9464 done
Copying blob d1ababb2c734 done
Copying config beba83a3b2 done
Writing manifest to image destination
Storing signatures
Error: OCI runtime error: error executing hook `/usr/bin/nvidia-container-toolkit` (exit code: 1)Here, simplecuda is just an oci-archive of docker.io/nvidia/cuda-ppc64le:10.2-base-centos7 (our HPC system uses IBM PowerPC).
The nvidia-container-toolkit.log looks like this
-- WARNING, the following logs are for debugging purposes only --
I0330 21:24:39.001988 1186667 nvc.c:282] initializing library context (version=1.3.0, build=16315ebdf4b9728e899f615e208b50c41d7a5d15)
I0330 21:24:39.002033 1186667 nvc.c:256] using root /
I0330 21:24:39.002038 1186667 nvc.c:257] using ldcache /etc/ld.so.cache
I0330 21:24:39.002043 1186667 nvc.c:258] using unprivileged user 65534:65534
I0330 21:24:39.002058 1186667 nvc.c:299] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0330 21:24:39.002241 1186667 nvc.c:301] dxcore initialization failed, continuing assuming a non-WSL environment
W0330 21:24:39.002259 1186667 nvc.c:167] skipping kernel modules load due to user namespace
I0330 21:24:39.002400 1186672 driver.c:101] starting driver service
E0330 21:24:39.002442 1186672 driver.c:161] could not start driver service: privilege change failed: operation not permitted
I0330 21:24:39.003214 1186667 driver.c:196] driver service terminated successfullyI've tried a whole variety of different Podman flag combinations mentioned earlier in this issue thread. None have worked. They all have the same errors above in the output and the log file.
I have the hook json file properly set up
% cat /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json
{
"version": "1.0.0",
"hook": {
"path": "/usr/bin/nvidia-container-toolkit",
"args": ["nvidia-container-toolkit", "prestart"],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
},
"when": {
"always": true,
"commands": [".*"]
},
"stages": ["prestart"]
}The nvidia-container-runtime config.toml looks like this
[76a@raptor07 ~]$ cat /etc/nvidia-container-runtime/config.toml
disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
#accept-nvidia-visible-devices-as-volume-mounts = false
[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
debug = "/tmp/.nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
no-cgroups = true
#user = "root:video"
ldconfig = "@/sbin/ldconfig"
[nvidia-container-runtime]
debug = "/tmp/.nvidia-container-runtime.log"My storage.conf looks like this
% cat ~/.config/containers/storage.conf
[storage]
driver = "overlay"
graphroot = "/tmp/subil-containers-peak"
rootless_storage_path = "$HOME/.local/share/containers/storage"
#rootless_storage_path = "/tmp/subil-containers-storage-peak"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
ignore_chown_errors = "true"
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,metacopy=on"
[storage.options.thinpool]For comparison, I also tested this on a PowerPC workstation (identical to the HPC nodes: IBM Power9 with Nvidia Tesla V100, RHEL 8.2) and it's the exact same errors there too. But once we set up the subuid/subgid mappings on the workstation and did echo “user.max_user_namespaces=28633” > /etc/sysctl.d/userns.conf, Podman was able to run the cuda container without issue.
[76a@raptor07 gpu]$ podman run --rm docker.io/nvidia/cuda-ppc64le:10.2-base-centos7 nvidia-smi -L
GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-4d2aad84-ad3d-430b-998c-6124d28d8e7c)So I know the issue is that we need both the subuid/subgid mappings and the user.max_user_namespaces. I want to know if it is possible to get the nvidia-container-toolkit working with rootless Podman without needing the subuid/subgid mappings.
For reference, we had a related issue (https://github.com/containers/podman/issues/8580) with MPI not working because of the lack of subuid/subgid mappings. @giuseppe was able to patch crun and Podman to make that work for Podman v3 and crun >=v0.17. I wanted to know if there was something that could be done here to make the nvidia-container-toolkit also work under the same conditions.
I'm happy to provide more details if you need.
zjuwyz
rhatdan
qhaas
elezar
giuseppe
daveman1010221
KCSesh
Ru13en
fuomag9
helson73
decandia50
baude
klueska
starry91
Issue or feature description rootless and rootful podman does not work with the nvidia plugin
Steps to reproduce the issue Install the nvidia plugin, configure it to run with podman execute the podman command and check if the devices is configured correctly.
Information to attach (optional if deemed irrelevant)
Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info Kernel version from uname -a Fedora 30 and later Any relevant kernel output lines from dmesg Driver information from nvidia-smi -a Docker version from docker version NVIDIA packages version from dpkg -l 'nvidia' or rpm -qa 'nvidia' NVIDIA container library version from nvidia-container-cli -V NVIDIA container library logs (see troubleshooting) Docker command, image and tag used
I am reporting this based on other users complaining. This is what they said.
We discovered that the ubuntu 18.04 machine needed a configuration change to get rootless working with nvidia: "no-cgroups = true" was set in /etc/nvidia-container-runtime/config.toml Unfortunately this config change did not work on Centos 7, but it did change the rootless error to: nvidia-container-cli: initialization error: cuda error: unknown error\\n\\"\""
This config change breaks podman running from root, with the error: Failed to initialize NVML: Unknown Error
Interestingly, root on ubuntu gets the same error even though rootless works.