slurmstepd: error: pyxis: mkdir: cannot create directory '/run/enroot': Permission denied

[kazuitox]

Hi,

I tried pyxis and got the following error:

[kazuito@mgmt ~]$ srun --container-image=centos grep PRETTY /etc/os-release slurmstepd: pyxis: importing docker image ... slurmstepd: error: pyxis: child 12004 failed with error code: 1 slurmstepd: error: pyxis: failed to import docker image slurmstepd: error: pyxis: printing contents of log file ... slurmstepd: error: pyxis: mkdir: cannot create directory '/run/enroot': Permission denied slurmstepd: pyxis: could not remove squashfs: No such file or directory slurmstepd: error: spank: required plugin spank_pyxis.so: task_init_privileged() failed with rc=-1 slurmstepd: error: spank_task_init_privileged failed slurmstepd: error: write to unblock task 0 failed: Broken pipe srun: error: vm-standard2-1-ad1-0001: task 0: Exited with exit code 1

It works fine without pyxis on compute nodes.

[kazuito@vm-standard2-1-ad1-0001 ~]$ enroot import docker://centos [kazuito@vm-standard2-1-ad1-0001 ~]$ enroot create --name centos centos.sqsh [kazuito@vm-standard2-1-ad1-0001 ~]$ enroot start --root --rw centos cat /etc/os-release

How do I solve this problem?

Thanks,

[flx42]

@kazuitox Apologies! This still need to be better documented. In our case, we use a custom enroot config in /etc/enroot/enroot.conf:

ENROOT_RUNTIME_PATH /run/enroot/user-$(id -u)
ENROOT_CACHE_PATH /raid/enroot-cache/group-$(id -g)
ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)

And then, in a Slurm prolog, we mkdir/chown these directories, for instance:

runtime_path="$(sudo -u "$SLURM_JOB_USER" sh -c 'echo "/run/enroot/user-$(id -u)"')"
mkdir -p "$runtime_path"
chown "$SLURM_JOB_UID:$(id -g "$SLURM_JOB_UID")" "$runtime_path"
chmod 0700 "$runtime_path"

This comes from the Slurm enroot DeepOps role: https://github.com/NVIDIA/deepops/blob/6b57cd1ccc80b74d671039b7609525ae70a9f8e8/roles/slurm-perf/templates/etc/slurm/prolog.d/50-all-enroot-dirs

[kazuitox]

@flx42 Thank you for the advice! I can run the pyxis!

[krono]

Sorry to dig this up again, but I seem to miss something.

I try to ad-hoc debug this issue on my site, but since we're not there yet with deploying prologs, I tired the following:

$ env \
 ENROOT_RUNTIME_PATH=/tmp/$(id -u) \
 ENROOT_DATA_PATH=/tmp/$(id -u) \
 ENROOT_CACHE_PATH=/tmp/$(id -u) \
 SLURM_DEBUG=2 \
 srun  -A $ACCOUNT -p $PARTITION -w $NODE --container-image=nvidia/cuda:11.4.1-base-ubuntu20.04 nvidia-smi

and got the result

srun: select/linear: init: Linear node selection plugin loaded with argument 20
srun: select/cray_aries: init: Cray/Aries node selection plugin loaded
srun: select/cons_res: common_init: select/cons_res loaded
srun: select/cons_tres: common_init: select/cons_tres loaded
srun: debug:  switch Cray/Aries plugin loaded.
srun: debug:  switch/none: init: switch NONE plugin loaded
srun: debug:  spank: opening plugin stack /run/slurm/conf/plugstack.conf
srun: debug:  /run/slurm/conf/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf"
srun: debug:  /run/slurm/conf/plugstack.conf: 2: include "/usr/share/pyxis/pyxis.conf"
srun: debug:  spank: opening plugin stack /usr/share/pyxis/pyxis.conf
srun: debug:  spank: /usr/share/pyxis/pyxis.conf:1: Loaded plugin spank_pyxis.so
srun: debug:  SPANK: appending plugin option "container-image"
srun: debug:  SPANK: appending plugin option "container-mounts"
srun: debug:  SPANK: appending plugin option "container-workdir"
srun: debug:  SPANK: appending plugin option "container-name"
srun: debug:  SPANK: appending plugin option "container-save"
srun: debug:  SPANK: appending plugin option "container-mount-home"
srun: debug:  SPANK: appending plugin option "no-container-mount-home"
srun: debug:  SPANK: appending plugin option "container-remap-root"
srun: debug:  SPANK: appending plugin option "no-container-remap-root"
srun: debug:  SPANK: appending plugin option "container-entrypoint"
srun: debug:  SPANK: appending plugin option "no-container-entrypoint"
srun: launch/slurm: init: launch Slurm plugin loaded
srun: debug:  mpi type = none
srun: debug:  propagating RLIMIT_CPU=18446744073709551615
srun: debug:  propagating RLIMIT_FSIZE=18446744073709551615
srun: debug:  propagating RLIMIT_DATA=18446744073709551615
srun: debug:  propagating RLIMIT_STACK=8388608
srun: debug:  propagating RLIMIT_CORE=0
srun: debug:  propagating RLIMIT_RSS=18446744073709551615
srun: debug:  propagating RLIMIT_NPROC=31621
srun: debug:  propagating RLIMIT_NOFILE=1024
srun: debug:  propagating RLIMIT_MEMLOCK=18446744073709551615
srun: debug:  propagating RLIMIT_AS=18446744073709551615
srun: debug:  propagating SLURM_PRIO_PROCESS=0
srun: debug:  propagating UMASK=0022
srun: debug:  Entering slurm_allocation_msg_thr_create()
srun: debug:  port from net_stream_listen is 35425
srun: debug:  Entering _msg_thr_internal
srun: debug:  auth/munge: init: Munge authentication plugin loaded
srun: Waiting for nodes to boot (delay looping 450 times @ 0.100000 secs x index)
srun: Nodes $NODE are ready for job
srun: jobid 5005: nodes(1):`$NODE' cpu counts: 2(x1)
srun: debug:  requesting job 5005, user 5811, nodes 1 including (a6k5-01)
srun: debug:  cpus 2, tasks 1, name nvidia-smi, relative 65534
srun: launch/slurm: launch_p_step_launch: CpuBindType=(null type)
srun: debug:  Entering slurm_step_launch
srun: debug:  mpi type = (null)
srun: debug:  mpi/none: p_mpi_hook_client_prelaunch: Using mpi/none
srun: debug:  Entering _msg_thr_create()
srun: debug:  initialized stdio listening socket, port 41519
srun: debug:  Started IO server thread (140202398373632)
srun: debug:  Entering _launch_tasks
srun: launching StepId=5005.0 on host $NODE, 1 tasks: 0
srun: route/default: init: route default plugin loaded
srun: debug:  launch returned msg_rc=0 err=0 type=8001
srun: launch/slurm: _task_start: Node a6k5-01, 1 tasks started
pyxis: importing docker image ...
slurmstepd-$NODE: error: pyxis: child 2103628 failed with error code: 1
slurmstepd-$NODE: error: pyxis: failed to import docker image
slurmstepd-$NODE: error: pyxis: printing contents of log file ...
slurmstepd-$NODE: error: pyxis:     mkdir: cannot create directory ‘/run/user/5811’: Permission denied
slurmstepd-$NODE: error: pyxis: couldn't start container
slurmstepd-$NODE: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd-$NODE: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd-$NODE: error: Failed to invoke spank plugin stack
srun: launch/slurm: _task_finish: Received task exit notification for 1 task of StepId=5005.0 (status=0x0100).
srun: error: $NODE: task 0: Exited with exit code 1
srun: debug:  task 0 done
srun: debug:  IO thread exiting
srun: debug:  Leaving _msg_thr_internal

I wonder where /run/user/... comes from. The Enroot-envvar is differen, it's not /run/pyxis (as per https://github.com/NVIDIA/pyxis/wiki/Setup) and also not /run/enroot as in @kazuitox case. (yes, XDG_RUNTIME_DIR is the /run/user/.. one, but Shouldn't the environment variable have been picked up?)

Note: I'm using slurm_pam_adopt and have hence removed pam_systemd. Am I missing something?

[krono]

Addition: the XDG_RUNTIME_DIR was picked up from the login node where I issued the srun command. In retrospect, that is what the manpage says, but it was a wee bit surprising…

Running (note the -i)

$ env -i\
 ENROOT_RUNTIME_PATH=/tmp/$(id -u) \
 ENROOT_DATA_PATH=/tmp/$(id -u) \
 ENROOT_CACHE_PATH=/tmp/$(id -u) \
 SLURM_DEBUG=2 \
 srun  -A $ACCOUNT -p $PARTITION -w $NODE --container-image=nvidia/cuda:11.4.1-base-ubuntu20.04 nvidia-smi

now gives

...
slurmstepd-$NODE: error: pyxis: child 2111469 failed with error code: 1
slurmstepd-$NODE: error: pyxis: failed to import docker image
slurmstepd-$NODE: error: pyxis: printing contents of log file ...
slurmstepd-$NODE error: pyxis:     mkdir: cannot create directory ‘/run/enroot’: Permission denied
slurmstepd-$NODE: error: pyxis: couldn't start container
slurmstepd-$NODE error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd-$NODE: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd-$NODE: error: Failed to invoke spank plugin stack

and doing a global mkdir /run/enroot makes things run.

Maybe I'd should open a new issue…

[flx42]

@krono please file a new issue :) The strategy for passing environment variables to enroot has changed in 0.11, it's now a deny-list: https://github.com/NVIDIA/pyxis/blob/47613beaf2a63f38b8e306119ab0df5163c1350d/pyxis_slurmstepd.c#L271-L291 It's to avoid users getting themselves into trouble with enroot failing if they try to change the paths.

But you can use ENROOT_RUNTIME_PATH=/tmp/$(id -u) in enroot.conf as the admin, it will be evaluated.

[krono]

Ah, reasonable!