Closed kazuitox closed 4 years ago
@kazuitox Apologies! This still need to be better documented.
In our case, we use a custom enroot config in /etc/enroot/enroot.conf
:
ENROOT_RUNTIME_PATH /run/enroot/user-$(id -u)
ENROOT_CACHE_PATH /raid/enroot-cache/group-$(id -g)
ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)
And then, in a Slurm prolog, we mkdir
/chown
these directories, for instance:
runtime_path="$(sudo -u "$SLURM_JOB_USER" sh -c 'echo "/run/enroot/user-$(id -u)"')"
mkdir -p "$runtime_path"
chown "$SLURM_JOB_UID:$(id -g "$SLURM_JOB_UID")" "$runtime_path"
chmod 0700 "$runtime_path"
This comes from the Slurm enroot DeepOps role: https://github.com/NVIDIA/deepops/blob/6b57cd1ccc80b74d671039b7609525ae70a9f8e8/roles/slurm-perf/templates/etc/slurm/prolog.d/50-all-enroot-dirs
@flx42 Thank you for the advice! I can run the pyxis!
Sorry to dig this up again, but I seem to miss something.
I try to ad-hoc debug this issue on my site, but since we're not there yet with deploying prologs, I tired the following:
$ env \
ENROOT_RUNTIME_PATH=/tmp/$(id -u) \
ENROOT_DATA_PATH=/tmp/$(id -u) \
ENROOT_CACHE_PATH=/tmp/$(id -u) \
SLURM_DEBUG=2 \
srun -A $ACCOUNT -p $PARTITION -w $NODE --container-image=nvidia/cuda:11.4.1-base-ubuntu20.04 nvidia-smi
and got the result
srun: select/linear: init: Linear node selection plugin loaded with argument 20
srun: select/cray_aries: init: Cray/Aries node selection plugin loaded
srun: select/cons_res: common_init: select/cons_res loaded
srun: select/cons_tres: common_init: select/cons_tres loaded
srun: debug: switch Cray/Aries plugin loaded.
srun: debug: switch/none: init: switch NONE plugin loaded
srun: debug: spank: opening plugin stack /run/slurm/conf/plugstack.conf
srun: debug: /run/slurm/conf/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf"
srun: debug: /run/slurm/conf/plugstack.conf: 2: include "/usr/share/pyxis/pyxis.conf"
srun: debug: spank: opening plugin stack /usr/share/pyxis/pyxis.conf
srun: debug: spank: /usr/share/pyxis/pyxis.conf:1: Loaded plugin spank_pyxis.so
srun: debug: SPANK: appending plugin option "container-image"
srun: debug: SPANK: appending plugin option "container-mounts"
srun: debug: SPANK: appending plugin option "container-workdir"
srun: debug: SPANK: appending plugin option "container-name"
srun: debug: SPANK: appending plugin option "container-save"
srun: debug: SPANK: appending plugin option "container-mount-home"
srun: debug: SPANK: appending plugin option "no-container-mount-home"
srun: debug: SPANK: appending plugin option "container-remap-root"
srun: debug: SPANK: appending plugin option "no-container-remap-root"
srun: debug: SPANK: appending plugin option "container-entrypoint"
srun: debug: SPANK: appending plugin option "no-container-entrypoint"
srun: launch/slurm: init: launch Slurm plugin loaded
srun: debug: mpi type = none
srun: debug: propagating RLIMIT_CPU=18446744073709551615
srun: debug: propagating RLIMIT_FSIZE=18446744073709551615
srun: debug: propagating RLIMIT_DATA=18446744073709551615
srun: debug: propagating RLIMIT_STACK=8388608
srun: debug: propagating RLIMIT_CORE=0
srun: debug: propagating RLIMIT_RSS=18446744073709551615
srun: debug: propagating RLIMIT_NPROC=31621
srun: debug: propagating RLIMIT_NOFILE=1024
srun: debug: propagating RLIMIT_MEMLOCK=18446744073709551615
srun: debug: propagating RLIMIT_AS=18446744073709551615
srun: debug: propagating SLURM_PRIO_PROCESS=0
srun: debug: propagating UMASK=0022
srun: debug: Entering slurm_allocation_msg_thr_create()
srun: debug: port from net_stream_listen is 35425
srun: debug: Entering _msg_thr_internal
srun: debug: auth/munge: init: Munge authentication plugin loaded
srun: Waiting for nodes to boot (delay looping 450 times @ 0.100000 secs x index)
srun: Nodes $NODE are ready for job
srun: jobid 5005: nodes(1):`$NODE' cpu counts: 2(x1)
srun: debug: requesting job 5005, user 5811, nodes 1 including (a6k5-01)
srun: debug: cpus 2, tasks 1, name nvidia-smi, relative 65534
srun: launch/slurm: launch_p_step_launch: CpuBindType=(null type)
srun: debug: Entering slurm_step_launch
srun: debug: mpi type = (null)
srun: debug: mpi/none: p_mpi_hook_client_prelaunch: Using mpi/none
srun: debug: Entering _msg_thr_create()
srun: debug: initialized stdio listening socket, port 41519
srun: debug: Started IO server thread (140202398373632)
srun: debug: Entering _launch_tasks
srun: launching StepId=5005.0 on host $NODE, 1 tasks: 0
srun: route/default: init: route default plugin loaded
srun: debug: launch returned msg_rc=0 err=0 type=8001
srun: launch/slurm: _task_start: Node a6k5-01, 1 tasks started
pyxis: importing docker image ...
slurmstepd-$NODE: error: pyxis: child 2103628 failed with error code: 1
slurmstepd-$NODE: error: pyxis: failed to import docker image
slurmstepd-$NODE: error: pyxis: printing contents of log file ...
slurmstepd-$NODE: error: pyxis: mkdir: cannot create directory ‘/run/user/5811’: Permission denied
slurmstepd-$NODE: error: pyxis: couldn't start container
slurmstepd-$NODE: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd-$NODE: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd-$NODE: error: Failed to invoke spank plugin stack
srun: launch/slurm: _task_finish: Received task exit notification for 1 task of StepId=5005.0 (status=0x0100).
srun: error: $NODE: task 0: Exited with exit code 1
srun: debug: task 0 done
srun: debug: IO thread exiting
srun: debug: Leaving _msg_thr_internal
I wonder where /run/user/...
comes from.
The Enroot-envvar is differen, it's not /run/pyxis
(as per https://github.com/NVIDIA/pyxis/wiki/Setup)
and also not /run/enroot
as in @kazuitox case.
(yes, XDG_RUNTIME_DIR is the /run/user/.. one, but Shouldn't the environment variable have been picked up?)
Note: I'm using slurm_pam_adopt
and have hence removed pam_systemd
.
Am I missing something?
Addition: the XDG_RUNTIME_DIR
was picked up from the login node where I issued the srun
command. In retrospect, that is what the manpage says, but it was a wee bit surprising…
Running (note the -i
)
$ env -i\
ENROOT_RUNTIME_PATH=/tmp/$(id -u) \
ENROOT_DATA_PATH=/tmp/$(id -u) \
ENROOT_CACHE_PATH=/tmp/$(id -u) \
SLURM_DEBUG=2 \
srun -A $ACCOUNT -p $PARTITION -w $NODE --container-image=nvidia/cuda:11.4.1-base-ubuntu20.04 nvidia-smi
now gives
...
slurmstepd-$NODE: error: pyxis: child 2111469 failed with error code: 1
slurmstepd-$NODE: error: pyxis: failed to import docker image
slurmstepd-$NODE: error: pyxis: printing contents of log file ...
slurmstepd-$NODE error: pyxis: mkdir: cannot create directory ‘/run/enroot’: Permission denied
slurmstepd-$NODE: error: pyxis: couldn't start container
slurmstepd-$NODE error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd-$NODE: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd-$NODE: error: Failed to invoke spank plugin stack
and doing a global mkdir /run/enroot
makes things run.
Maybe I'd should open a new issue…
@krono please file a new issue :) The strategy for passing environment variables to enroot has changed in 0.11, it's now a deny-list: https://github.com/NVIDIA/pyxis/blob/47613beaf2a63f38b8e306119ab0df5163c1350d/pyxis_slurmstepd.c#L271-L291 It's to avoid users getting themselves into trouble with enroot failing if they try to change the paths.
But you can use ENROOT_RUNTIME_PATH=/tmp/$(id -u)
in enroot.conf
as the admin, it will be evaluated.
Ah, reasonable!
Hi,
I tried pyxis and got the following error:
[kazuito@mgmt ~]$ srun --container-image=centos grep PRETTY /etc/os-release slurmstepd: pyxis: importing docker image ... slurmstepd: error: pyxis: child 12004 failed with error code: 1 slurmstepd: error: pyxis: failed to import docker image slurmstepd: error: pyxis: printing contents of log file ... slurmstepd: error: pyxis: mkdir: cannot create directory '/run/enroot': Permission denied slurmstepd: pyxis: could not remove squashfs: No such file or directory slurmstepd: error: spank: required plugin spank_pyxis.so: task_init_privileged() failed with rc=-1 slurmstepd: error: spank_task_init_privileged failed slurmstepd: error: write to unblock task 0 failed: Broken pipe srun: error: vm-standard2-1-ad1-0001: task 0: Exited with exit code 1
It works fine without pyxis on compute nodes.
[kazuito@vm-standard2-1-ad1-0001 ~]$ enroot import docker://centos [kazuito@vm-standard2-1-ad1-0001 ~]$ enroot create --name centos centos.sqsh [kazuito@vm-standard2-1-ad1-0001 ~]$ enroot start --root --rw centos cat /etc/os-release
How do I solve this problem?
Thanks,