We are using Pyxis & Enroot on Oracle Linux 7.9 and we are getting an errors.
Environment:
Baremetal 8xA100
PRETTY_NAME="Oracle Linux Server 7.9"
Kernel: 5.4.17-2036.100.6.1.el7uek.x86_64
NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0
slurm-20.11.2-1.el7.x86_64
Error:
After applying Pyxis, we did a job test. However, namespace permission error occurred.
Problem state
$ srun --container-image=/nfs/cluster/pyxis_test/enroot_image/centos.sqsh grep PRETTY /etc/os-release
slurmstepd: error: pyxis: container start failed with error code: 1
slurmstepd: error: pyxis: printing contents of log file ...
slurmstepd: error: pyxis: enroot-nsenter: failed to create user namespace: Permission denied
slurmstepd: error: pyxis: couldn't start container
slurmstepd: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd: error: Failed to invoke spank plugin stack
srun: error: inst-foysz-vocal-drake: task 0: Exited with exit code 1
Check the current settings
$ ./enroot-check_*.run --verify
Kernel version:
Linux version 5.4.17-2036.100.6.1.el7uek.x86_64 (mockbuild@jenkins-172-17-0-2-980b5770-aa59-43ca-ab82-ac28f1d4376e) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39.0.3) (GCC)) #2 SMP Thu Oct 29 17:04:48 PDT 2020
Kernel configuration:
CONFIG_NAMESPACES : OK
CONFIG_USER_NS : OK
CONFIG_SECCOMP_FILTER : OK
CONFIG_OVERLAY_FS : OK (module)
CONFIG_X86_VSYSCALL_EMULATION : OK
CONFIG_VSYSCALL_EMULATE : KO (required if glibc <= 2.13)
CONFIG_VSYSCALL_NATIVE : KO (required if glibc <= 2.13)
Kernel command line:
vsyscall=native : KO (required if glibc <= 2.13)
vsyscall=emulate : KO (required if glibc <= 2.13)
Kernel parameters:
user.max_user_namespaces : OK
user.max_mnt_namespaces : OK
Extra packages:
nvidia-container-cli : OK
Kernel parameters updated and did a reboot. After reboot:
Investigation:
I started looking at enroot source code and found bundle.sh which has logic to validate if the kernel parameters were set. https://github.com/NVIDIA/enroot/blob/master/src/bundle.sh#L145:L155
You will see it only checks for “centos7|rhel7“. , should this be updated to add Oracle Linux like this ? centos7|rhel7|ol7”
On OCI cloud
source /etc/os-release 2> /dev/null; echo "${ID-}${VERSION_ID-}"
ol7.9
I did a sample “enroot” test like below and it worked fine with the default bundle.sh and with bundle.sh modified to include “ol7*” and both times it worked.
I ran this sample example. This is standalone, so it helps to isolate enroot issue from slurm or slurm pyxis invoking enroot.
Usage example:
# Import and start an Ubuntu image from DockerHub
$ enroot import docker://ubuntu
$ enroot create ubuntu.sqsh
$ enroot start ubuntu
I am not an enroot or pyxis expert, but I am assuming, since standalone enroot works, the issue could be in how pyxis is configured to use enroot or pyxis follows a code path which is different from the above simple example.
How was pyxis installed and configured?
Since RPM package is not currently provided, for Oracle Linux the source installation was used. From Pyxis github, clone source and build.
Github's installation guide specifies the path of the spank_pyxis.so library built in /etc/slurm/plugstack.conf.d/pyxis.conf to recognize the Spank plugin specified by Slurm, but this path is set well in Slurm, it don't seem to recognize it. As an alternative, create a plugstack.conf file that explicitly defines the Spank Plugin library in the slurm config directory and specify the library path of spank_pyxis.so.
Install Guide in Github ← have a problem
# vi /etc/slurm/plugstack.conf
required /usr/local/lib/slurm/spank_pyxis.so
# systemctl restart slurmctld && systemctl status slurmctld
Worker Nodes
# for i in {1..2}; do ssh node$i "mkdir /usr/local/lib/slurm"; done
# for i in {1..2}; do scp /usr/local/lib/slurm/spank_pyxis.so
node$i:/usr/local/lib/slurm/; done
# for i in {1..2}; do scp /etc/slurm/plugstack.conf node$i:/etc/slurm/; done
# for i in {1..4}; do ssh node$i "systemctl restart slurmd && systemctl status slurmd";
done
Had to change the PATH of ENROOT from /run to /tmp , otherwise it fails with
slurmstepd: error: pyxis: mkdir: cannot create directory ‘/run/enroot’: Permission denied
Hello Pyxis team ,
We are using Pyxis & Enroot on Oracle Linux 7.9 and we are getting an errors.
Environment: Baremetal 8xA100 PRETTY_NAME="Oracle Linux Server 7.9" Kernel: 5.4.17-2036.100.6.1.el7uek.x86_64 NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 slurm-20.11.2-1.el7.x86_64
Error: After applying Pyxis, we did a job test. However, namespace permission error occurred. Problem state
Check the current settings
Kernel parameters updated and did a reboot. After reboot:
Investigation: I started looking at enroot source code and found bundle.sh which has logic to validate if the kernel parameters were set.
https://github.com/NVIDIA/enroot/blob/master/src/bundle.sh#L145:L155 You will see it only checks for “centos7|rhel7“. , should this be updated to add Oracle Linux like this ? centos7|rhel7|ol7”
On OCI cloud source /etc/os-release 2> /dev/null; echo "${ID-}${VERSION_ID-}" ol7.9
I did a sample “enroot” test like below and it worked fine with the default bundle.sh and with bundle.sh modified to include “ol7*” and both times it worked.
I ran this sample example. This is standalone, so it helps to isolate enroot issue from slurm or slurm pyxis invoking enroot.
Usage example:
I am not an enroot or pyxis expert, but I am assuming, since standalone enroot works, the issue could be in how pyxis is configured to use enroot or pyxis follows a code path which is different from the above simple example.
How was pyxis installed and configured?
Since RPM package is not currently provided, for Oracle Linux the source installation was used. From Pyxis github, clone source and build.
Github's installation guide specifies the path of the spank_pyxis.so library built in /etc/slurm/plugstack.conf.d/pyxis.conf to recognize the Spank plugin specified by Slurm, but this path is set well in Slurm, it don't seem to recognize it. As an alternative, create a plugstack.conf file that explicitly defines the Spank Plugin library in the slurm config directory and specify the library path of spank_pyxis.so. Install Guide in Github ← have a problem
We changed install guide ← works well Master node
Had to change the PATH of ENROOT from /run to /tmp , otherwise it fails with slurmstepd: error: pyxis: mkdir: cannot create directory ‘/run/enroot’: Permission denied