Closed coolinger closed 3 years ago
Hi Quirin,
It's possible that the approach we use in pyxis won't work for a NFS file handle. I will try to test when I have time.
While ENROOT_CACHE_PATH
can probably be used on a NFS (to share pulled container layers), the other ones are a bit risky. For instance, sharing ENROOT_DATA_PATH
will create conflicts when a multi-node job tries to create a container named "foo" on all its nodes at the same time.
In our case, we use a custom enroot config in /etc/enroot/enroot.conf
:
ENROOT_RUNTIME_PATH /run/enroot/user-$(id -u)
ENROOT_CACHE_PATH /raid/enroot-cache/group-$(id -g)
ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)
And then, in a Slurm prolog, we mkdir/chown these directories (to workaround the fact that /run/user/
runtime_path="$(sudo -u "$SLURM_JOB_USER" sh -c 'echo "/run/enroot/user-$(id -u)"')"
mkdir -p "$runtime_path"
chown "$SLURM_JOB_UID:$(id -g "$SLURM_JOB_UID")" "$runtime_path"
chmod 0700 "$runtime_path"
This comes from the Slurm enroot DeepOps role: https://github.com/NVIDIA/deepops/blob/6b57cd1ccc80b74d671039b7609525ae70a9f8e8/roles/slurm-perf/templates/etc/slurm/prolog.d/50-all-enroot-dirs
In one of our cluster, we have plenty of RAM so /tmp
is a tmpfs so it makes all container operations super fast. In other cases, we store the container filesystems (ENROOT_DATA_PATH
) to the local RAID array. It is slower but doesn't take RAM.
Thanks for the reply. The prolog scripts are a very welcome inspiration. Our cluster is set up with netboot nodes with enough ram, combining their local SSDs into a Ceph cluster and using CephFS. I will set up enroot now that it uses the prolog-created runtime, the data in a per user cephfs and a nfs shared cache. I will strongly suggest the users to create the containers using enroot before running on slurm, and only use --container-name in srun.
I will strongly suggest the users to create the containers using enroot before running on slurm, and only use --container-name in srun.
Why? Do you want to have a single container filesystem across all nodes?
My idea was to use enroot/pyxis as a provider of different runtime environments for the user's code if needed. Like old code needing Ubuntu 16.04 libraries and new code needing 20.04, whereas the slurm nodes are all running 18.04 right now. The code itself is in the user's $HOME, which is mounted into the container. As I understood enroot initially, the containers would be a chroot like system, and with them being read-only, there should be no problems running the same container from multiple nodes? I somehow assumed the containers itself to use an overlayfs and not be persistent about changes in the rootfs...
As I understood enroot initially, the containers would be a chroot like system, and with them being read-only, there should be no problems running the same container from multiple nodes?
It was not exactly designed for this use case. The enroot "hooks" might become confused here.
In our case we have one copy of the container filesystem per node. And it's possible to store a squashfs image on a shared filesystem and pass it to pyxis, but it will still be extracted locally. Our use case does require writing to the container filesystem.
I somehow assumed the containers itself to use an overlayfs and not be persistent about changes in the rootfs...
This is possible with enroot by doing enroot start
of a squashfs file directly, without an enroot create
:
https://github.com/NVIDIA/enroot/blob/master/doc/cmd/start.md#starting-container-images
This is not exposed in pyxis today. Looks like directly using enroot might be a better fit for your use case as of today.
It was not exactly designed for this use case. The enroot "hooks" might become confused here.
Yep. That happened just now when the usage started to rise and a second job was started on the same node.
This is possible with enroot by doing
enroot start
of a squashfs file directly, without anenroot create
: https://github.com/NVIDIA/enroot/blob/master/doc/cmd/start.md#starting-container-images This is not exposed in pyxis today. Looks like directly using enroot might be a better fit for your use case as of today.
I switched to do that now.
Thanks for your time, keep up the good work. Should I open an issue as feature request for my use case?
Thanks for your time, keep up the good work. Should I open an issue as feature request for my use case?
Sure, why not! I don't make any guarantee when (if ever!) we'll get to it. We'll see how our needs evolve internally.
Thanks
Hi,
running with NFS $HOME. Since slurm is running with cgroups and without logind, I had to change the ENROOT_DIRs in /etc/enroot/enroot.conf:
enroot alone works:
pyxis does not work:
If I move the ${HOME}/enroot/data to a non-fs path and symlink it, pyxis works.
But since enroot itself works inside the NFS path, I guess this is an issue with pyxis and hope for a possible solution.
Thanks Quirin