Closed biocyberman closed 4 years ago
I changed two lines in the common.h to these:
#define PYXIS_RUNTIME_PATH "/tmp/pyxis"
#define PYXIS_USER_RUNTIME_PATH "/tmp/pyxis/%d"
Make install again, I could get a bit further:
srun --container-image=centos grep NAME /etc/os-release
srun: job 323 queued and waiting for resources
srun: job 323 has been allocated resources
slurmstepd: pyxis: running "enroot import" ...
slurmstepd: error: pyxis: child 33628 failed with error code: 1
slurmstepd: error: pyxis: enroot import failed
slurmstepd: pyxis: printing contents of log file ...
slurmstepd: error: pyxis: mktemp: failed to create directory via template ‘/scratch/enroot/tmp/10005/enroot.XXXXXXXXXX’: No such file or directory
slurmstepd: error: spank: required plugin spank_pyxis.so: task_init_privileged() failed with rc=-1
slurmstepd: error: spank_task_init_privileged failed
This is because I have ENROOT_TEMP_PATH
set to /scratch/enroot/tmp/${UID}". Changing it to
/scratch/enroot` and I could get this:
srun --container-image=centos grep NAME /etc/os-release
srun: job XXXX queued and waiting for resources
srun: job XXXX has been allocated resources
slurmstepd: pyxis: running "enroot import" ...
slurmstepd: pyxis: running "enroot create" ...
slurmstepd: pyxis: running "enroot start" ...
slurmstepd: error: xcpuinfo_hwloc_topo_load: failed (load will be required after read failures).
NAME="CentOS Linux"
PRETTY_NAME="CentOS Linux 7 (Core)"
CPE_NAME="cpe:/o:centos:centos:7"
So it seems working, but it is a tweak at the moment. And still wonder what does this error mean:
slurmstepd: error: xcpuinfo_hwloc_topo_load: failed (load will be required after read failures).
Hello @biocyberman, thanks for the bug report!
The following function should be called in the slurmd context (as root) and it will create /run/pyxis
:
https://github.com/NVIDIA/pyxis/blob/master/pyxis_slurmd.c#L17-L42
Could you check if this directory exists before the issue arises? If yes, what are the permissions on this folder? It it doesn't exist, you might need to restart the slurmd
service for this function to be called. Looks like I need to add this to the documentation.
Afterwards, for each job, the plugin will attempt to create directory /run/pyxis/<uid>
here:
https://github.com/NVIDIA/pyxis/blob/master/pyxis_slurmstepd.c#L353-L379
As the comment indicates, this is done as root and then chowned to the user. This is the part that is failing for you. If restarting slurmd
doesn't solve it, it might be a permission issue.
But you are right that we should probably make it configurable, since any directory with the sticky bit set would work.
Also slurmstepd: error: xcpuinfo_hwloc_topo_load: failed (load will be required after read failures).
is probably unrelated to pyxis, I've never seen that on my side.
@biocyberman I've documented the missing step (restarting slurmd
), thanks for the bug report!
After some modifications from installation guide, pyxis seems to work:
However, this command failed:
Slurm log in
/var/log/slurmd.log
on the compute node has these lines:/run
is owned by root and not writable by normal users. So, should there be a configuration option to set the RUNTIME_PATH to somewhere else?https://github.com/NVIDIA/pyxis/blob/a521ac06839660cebe482d4cf1a16792e8815f8f/common.h#L7