Closed BlueCloudDev closed 3 years ago
Hi @BlueCloudDev,
It might not be a hang from pyxis here, I agree the logging is confusing here. It prints the start of the image import, but it doesn't print when the image import is done and the container is started.
Can you check which enroot
processes are running on this node when it happens? I'm interested to check whether it's indeed stuck in enroot import
, or perhaps in enroot start
.
The nvcr.io images also ship with an entrypoint that can take some time to execute (but shouldn't cause a hang), it should be visible when looking at running processes. You can disable the entrypoint with an enroot hook:
$ cat /etc/enroot/mounts.d/90-entrypoint.fstab
/etc/enroot/entrypoint /etc/rc.local none x-create=file,bind,ro,nosuid,nodev,noexec,nofail,silent
$ cat /etc/enroot/entrypoint
if [ $# -gt 0 ]; then
exec "$@"
else
exec '/bin/bash'
fi
I may need some hand-holding here since I haven't worked with pyxis/enroot before. What steps do I have to take in order to check which enroot processes are running on the node?
[opc@inst-rmdks-modest-cobra ~]$ cat /etc/enroot/mounts.d/90-entrypoint.fstab cat: /etc/enroot/mounts.d/90-entrypoint.fstab: No such file or directory [opc@inst-rmdks-modest-cobra ~]$ cat /etc/enroot/mounts.d/ 10-system.fstab 20-config.fstab
[opc@inst-rmdks-modest-cobra ~]$ cat /etc/enroot/ enroot.conf enroot.conf.original environ.d/ hooks.d/ mounts.d/
I'm not seeing these folders/files
enroot list
returns nothing on both the control and worker nodes
What steps do I have to take in order to check which enroot processes are running on the node?
Try ps ax | grep enroot
, or check manually with htop
.
e.g. when pyxis is importing the image:
$ ps ax | grep enroot
36801 pts/2 S+ 0:00 bash /usr/bin/enroot import --output /run/pyxis/10009/1249239.1.squashfs docker://nvcr.io#nvidia/pytorch:21.02-py3
36826 pts/2 S+ 0:00 bash /usr/bin/enroot import --output /run/pyxis/10009/1249239.1.squashfs docker://nvcr.io#nvidia/pytorch:21.02-py3
38116 pts/2 R+ 0:00 rm --one-file-system --preserve-root -rf /tmp/enroot.aYZIBfBB3y
I'm not seeing these folders/files
Yeah, I was saying you need to add these files if you want to disable entrypoints from the nvcr.io image. But let's take a look at the running processes first.
74612 ? S 0:00 bash /usr/bin/enroot import --output /run/pyxis/1000/297.0.squashfs docker://nvcr.io/nvidia/pytorch:20.12-py3
74626 ? S 0:00 bash /usr/bin/enroot import --output /run/pyxis/1000/297.0.squashfs docker://nvcr.io/nvidia/pytorch:20.12-py3
75216 ? Sl 0:30 enroot-mksquashovlfs /tmp/enroot.d6X4vcQhZd/rootfs /run/pyxis/1000/297.0.squashfs -all-root -no-progress -processors 2 -comp lzo -noD
76204 pts/2 S+ 0:00 grep --color=auto enroot
Looks like it is indeed running. How long is this process expected to take? I've left it running for >1 hr and the command doesn't progress past the step I highlighted. I've noticed that it creates files in /tmp by default, does this process need to repeat every time a job is run?
I see that it is using only 2 processes here: -processors 2
. Perhaps it is running under a Slurm job allocation with only 2 cores, which could slow the import compared to running enroot import
outside of pyxis. But even with 2 cores being used, it should not take 1 hour.
I've noticed that it creates files in /tmp by default, does this process need to repeat every time a job is run?
This depends on your configuration in enroot.conf
, here is what we have:
ENROOT_RUNTIME_PATH /run/enroot/user-$(id -u)
ENROOT_CACHE_PATH /raid/enroot-cache/group-$(id -g)
ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)
Where /run and /tmp are tmpfs (we have enough RAM), and /raid is a RAID0 ext4 filesystem.
Some questions to continue investigating:
enroot import --output /run/pyxis/test.squashfs docker://nvcr.io/nvidia/pytorch:20.12-py3
on a compute node but outside a Slurm allocation? That should use all the available cores, and will be outside of pyxis.enroot
are you using?As far as CPU utilization goes, on the node I'm not seeing much at all. This is a 64-core 8x A100 system and next to nothing showing up on htop while running through slurm.
I modified the command you gave to account for the authentication with $oauthtoken.
[opc@inst-rmdks-modest-cobra tmp]$ sudo enroot import --output /run/pyxis/test.squashfs docker://$oauthtoken@nvcr.io#nvidia/pytorch:20.12-py3
[INFO] Querying registry for permission grant
[INFO] Authenticating with user: <anonymous>
[INFO] Authentication succeeded
[INFO] Fetching image manifest list
[INFO] Fetching image manifest
[INFO] Downloading 59 missing layers...
100% 59:0=0s 76c8b977361605981037d3f2290e042752b13007bcbadf39c0d6af61dd01675c
[INFO] Extracting image layers...
100% 58:0=0s 6a5697faee43339ef8e33e3839060252392ad99325a48f7c9d7e93c22db4d4cf
[INFO] Converting whiteouts...
100% 58:0=0s 6a5697faee43339ef8e33e3839060252392ad99325a48f7c9d7e93c22db4d4cf
[INFO] Creating squashfs filesystem...
Parallel mksquashfs: Using 128 processors
Creating 4.0 filesystem on /run/pyxis/test.squashfs, block size 131072.
[=========================================================================================================================|] 245848/245848 100%
Exportable Squashfs 4.0 filesystem, lzo compressed, data block size 131072
uncompressed data, compressed metadata, compressed fragments, compressed xattrs
duplicates are removed
Filesystem size 12754798.54 Kbytes (12455.86 Mbytes)
92.89% of uncompressed filesystem size (13730493.38 Kbytes)
Inode table size 2150147 bytes (2099.75 Kbytes)
32.17% of uncompressed inode table size (6682988 bytes)
Directory table size 2200012 bytes (2148.45 Kbytes)
43.00% of uncompressed directory table size (5116464 bytes)
Number of duplicate files found 28404
Number of inodes 178050
Number of files 149503
Number of fragments 8510
Number of symbolic links 4261
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 24286
Number of ids (unique uids + gids) 1
Number of uids 1
root (0)
Number of gids 1
root (0)
The process executed quickly
Enroot version
[opc@inst-rmdks-modest-cobra tmp]$ enroot version
3.2.0
Slurm version
[opc@inst-rmdks-modest-cobra tmp]$ sinfo --version
slurm 20.11.5
So with enroot directly you are using more cores indeed:
Parallel mksquashfs: Using 128 processors
I would recommend giving all cores to your Slurm job, and trying again with pyxis, and then check if the result is any different.
Also, when the container import seems hang, can you monitor the size of the squashfs file being created? e.g. in your case above, please check the size of /run/pyxis/1000/297.0.squashfs
over time. @3XX0 and myself are thinking it might be increasing very slowly.
You could also try with ENROOT_SQUASH_OPTIONS -noI -noD -noF -noX -no-duplicates
in your enroot.conf
, this should make the import faster, at the cost of more storage space.
Sharing your whole enroot.conf
could also be useful here.
enroot.conf - mostly empty
[opc@inst-rmdks-modest-cobra tmp]$ cat /etc/enroot/enroot.conf
#ENROOT_LIBRARY_PATH /usr/lib/enroot
#ENROOT_SYSCONF_PATH /etc/enroot
ENROOT_RUNTIME_PATH /tmp/enroot/user-$(id -u)
#ENROOT_CONFIG_PATH ${XDG_CONFIG_HOME}/enroot
#ENROOT_CACHE_PATH ${XDG_CACHE_HOME}/enroot
ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)
#ENROOT_TEMP_PATH ${TMPDIR:-/tmp}
# Gzip program used to uncompress digest layers.
#ENROOT_GZIP_PROGRAM gzip
# Options passed to zstd to compress digest layers.
#ENROOT_ZSTD_OPTIONS -1
# Options passed to mksquashfs to produce container images.
#ENROOT_SQUASH_OPTIONS -comp lzo -noD
# Make the container root filesystem writable by default.
#ENROOT_ROOTFS_WRITABLE no
# Remap the current user to root inside containers by default.
#ENROOT_REMAP_ROOT no
# Maximum number of processors to use for parallel tasks (0 means unlimited).
#ENROOT_MAX_PROCESSORS $(nproc)
# Maximum number of concurrent connections (0 means unlimited).
#ENROOT_MAX_CONNECTIONS 10
# Maximum time in seconds to wait for connections establishment (0 means unlimited).
#ENROOT_CONNECT_TIMEOUT 30
# Maximum time in seconds to wait for network operations to complete (0 means unlimited).
#ENROOT_TRANSFER_TIMEOUT 0
# Number of times network operations should be retried.
#ENROOT_TRANSFER_RETRIES 0
# Use a login shell to run the container initialization.
#ENROOT_LOGIN_SHELL yes
# Allow root to retain his superuser privileges inside containers.
#ENROOT_ALLOW_SUPERUSER no
# Use HTTP for outgoing requests instead of HTTPS (UNSECURE!).
#ENROOT_ALLOW_HTTP no
# Include user-specific configuration inside bundles by default.
#ENROOT_BUNDLE_ALL no
# Generate an embedded checksum inside bundles by default.
#ENROOT_BUNDLE_CHECKSUM no
# Mount the current user's home directory by default.
#ENROOT_MOUNT_HOME no
# Restrict /dev inside the container to a minimal set of devices.
#ENROOT_RESTRICT_DEV no
# Always use --force on command invocations.
#ENROOT_FORCE_OVERRIDE no
# SSL certificates settings:
#SSL_CERT_DIR
#SSL_CERT_FILE
# Proxy settings:
#all_proxy
#no_proxy
#http_proxy
#https_proxy
[opc@modest-cobra-bastion examples]$ srun --cpus-per-task=128 --container-image "nvcr.io/nvidia/pytorch:20.12-py3" grep PRETTY /etc/os-release
pyxis: importing docker image ...
PRETTY_NAME="Ubuntu 20.04.1 LTS"
This works, seems to have been limited by the number of CPUs getting set to 2 somewhere. Where should default resource limits be configured? Slurm or Pyxis?
This works, seems to have been limited by the number of CPUs getting set to 2 somewhere. Where should default resource limits be configured? Slurm or Pyxis?
It's how the Slurm job (or Slurm partition) was configured, Slurm likely created a cpuset cgroup to limit which cores can be accessed by your process. For example:
$ srun -l --ntasks-per-node=1 grep Cpus_allowed_list /proc/self/status
0: Cpus_allowed_list: 0,6
Here it's 2 logical cores (1 physical core). But if the partition is set to exclusive, you would see all the cores being available. Pyxis uses all the cores allocated to the Slurm job.
I also recommend to set ENROOT_SQUASH_OPTIONS -noI -noD -noF -noX -no-duplicates
in enroot.conf
, it should speed up the process even if you have only 2 cores available.
I think this issue is now resolved, closing. Please reopen this bug or open a new one if you have other questions on this topic.
[opc@modest-cobra-bastion examples]$ srun --container-image=centos grep PRETTY /etc/os-release pyxis: importing docker image ... PRETTY_NAME="CentOS Linux 8"
[opc@modest-cobra-bastion examples]$ srun --container-image "nvcr.io/nvidia/pytorch:20.12-py3" grep PRETTY /etc/os-release pyxis: importing docker image ...
The command sits indefinitely. I've left it for up to 20 minutes without any change. I can't find any network traffic on the node that would indicate that it is downloading, nor any CPU activity that would indicate other work related to the image. The
.credentials
file is set as:[opc@inst-rmdks-modest-cobra tmp]$ cat ~/enroot/.credentials
NVIDIA GPU Cloud (both endpoints are required)
machine nvcr.io login $oauthtoken password
machine authn.nvidia.com login $oauthtoken password