search

sylabs / singularity

SingularityCE is the Community Edition of Singularity, an open source container platform designed to be simple, fast, and secure.
https://sylabs.io/docs/
Other
765 stars 98 forks source link

"permission denied" error when exec-ing non-exectuable is not friendly #607

Open marc-rigter opened 2 years ago

marc-rigter commented 2 years ago

Version of Singularity 3.7.3 I also get the same behaviour with version 3.9.

Describe the bug When I try to execute the container using "sudo singularity exec --bind /home/marcrigter/github/mopo:/home/mopo mopo.sif /home/mopo/scripts/test.sh" I get an error "StageTwo() permission denied" and no other debug information.

Here is the output when using the -d flag:

DEBUG [U=0,P=109209] persistentPreRun() Singularity version: 3.7.3 DEBUG [U=0,P=109209] persistentPreRun() Parsing configuration file /usr/local/etc/singularity/singularity.conf DEBUG [U=0,P=109209] handleConfDir() /root/.singularity already exists. Not creating. DEBUG [U=0,P=109209] execStarter() Saving umask 0022 for propagation into container DEBUG [U=0,P=109209] execStarter() Checking for encrypted system partition DEBUG [U=0,P=109209] Init() Image format detection DEBUG [U=0,P=109209] Init() Check for sandbox image format DEBUG [U=0,P=109209] Init() sandbox format initializer returned: not a directory image DEBUG [U=0,P=109209] Init() Check for sif image format DEBUG [U=0,P=109209] Init() sif image format detected DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LANGUAGE environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LANG environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LS_COLORS environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding TERM environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding MAIL environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LOGNAME environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding USER environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SHELL environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_COMMAND environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_USER environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_UID environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_GID environment variable DEBUG [U=0,P=109209] SetContainerEnv() Forwarding USER_PATH environment variable VERBOSE [U=0,P=109209] SetContainerEnv() Setting HOME=/root VERBOSE [U=0,P=109209] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DEBUG [U=0,P=109209] init() Use starter binary /usr/local/libexec/singularity/bin/starter VERBOSE [U=0,P=109209] print() Set messagelevel to: 5 VERBOSE [U=0,P=109209] init() Starter initialization DEBUG [U=0,P=109209] load_overlay_module() Trying to load overlay kernel module DEBUG [U=0,P=109209] load_overlay_module() Overlay seems supported by the kernel VERBOSE [U=0,P=109209] is_suid() Check if we are running as setuid DEBUG [U=0,P=109209] read_engine_config() Read engine configuration DEBUG [U=0,P=109209] init() Wait completion of stage1 DEBUG [U=0,P=109220] set_parent_death_signal() Set parent death signal to 9 VERBOSE [U=0,P=109220] init() Spawn stage 1 DEBUG [U=0,P=109220] startup() singularity runtime engine selected VERBOSE [U=0,P=109220] startup() Execute stage 1 DEBUG [U=0,P=109220] StageOne() Entering stage 1 DEBUG [U=0,P=109220] prepareRootCaps() Root full capabilities DEBUG [U=0,P=109220] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for user bind path /home/mrigter/github/mopo: no mount point DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for home directory /root: no mount point DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for current working directory /home/mrigter/github/mopo/singularity: no mount point DEBUG [U=0,P=109220] Init() Image format detection DEBUG [U=0,P=109220] Init() Check for sandbox image format DEBUG [U=0,P=109220] Init() sandbox format initializer returned: not a directory image DEBUG [U=0,P=109220] Init() Check for sif image format DEBUG [U=0,P=109220] Init() sif image format detected DEBUG [U=0,P=109220] setSessionLayer() Overlay seems supported and allowed by kernel DEBUG [U=0,P=109220] setSessionLayer() Attempting to use overlayfs (enable overlay = try) VERBOSE [U=0,P=109209] wait_child() stage 1 exited with status 0 DEBUG [U=0,P=109209] cleanup_fd() Close file descriptor 4 DEBUG [U=0,P=109209] cleanup_fd() Close file descriptor 5 DEBUG [U=0,P=109209] cleanup_fd() Close file descriptor 6 DEBUG [U=0,P=109209] init() Set child signal mask DEBUG [U=0,P=109209] init() Create socketpair for master communication channel DEBUG [U=0,P=109209] init() Create RPC socketpair for communication between stage 2 and RPC server VERBOSE [U=0,P=109209] init() Spawn master process DEBUG [U=0,P=109226] set_parent_death_signal() Set parent death signal to 9 VERBOSE [U=0,P=109226] create_namespace() Create mount namespace VERBOSE [U=0,P=109209] enter_namespace() Entering in mount namespace DEBUG [U=0,P=109209] enter_namespace() Opening namespace file ns/mnt VERBOSE [U=0,P=109226] create_namespace() Create mount namespace VERBOSE [U=0,P=109232] init() Spawn RPC server DEBUG [U=0,P=109209] startup() singularity runtime engine selected VERBOSE [U=0,P=109209] startup() Execute master process DEBUG [U=0,P=109232] startup() singularity runtime engine selected VERBOSE [U=0,P=109232] startup() Serve RPC requests DEBUG [U=0,P=109209] setupSessionLayout() Using Layer system: overlay DEBUG [U=0,P=109209] setupOverlayLayout() Creating overlay SESSIONDIR layout DEBUG [U=0,P=109209] addRootfsMount() Mount rootfs in read-only mode DEBUG [U=0,P=109209] addRootfsMount() Image type is 4096 DEBUG [U=0,P=109209] addRootfsMount() Mounting block [squashfs] image: /home/mrigter/github/mopo/singularity/mopo.sif DEBUG [U=0,P=109209] addKernelMount() Checking configuration file for 'mount proc' DEBUG [U=0,P=109209] addKernelMount() Adding proc to mount list VERBOSE [U=0,P=109209] addKernelMount() Default mount: /proc:/proc DEBUG [U=0,P=109209] addKernelMount() Checking configuration file for 'mount sys' DEBUG [U=0,P=109209] addKernelMount() Adding sysfs to mount list VERBOSE [U=0,P=109209] addKernelMount() Default mount: /sys:/sys DEBUG [U=0,P=109209] addDevMount() Checking configuration file for 'mount dev' DEBUG [U=0,P=109209] addDevMount() Adding dev to mount list VERBOSE [U=0,P=109209] addDevMount() Default mount: /dev:/dev DEBUG [U=0,P=109209] addHostMount() Not mounting host file systems per configuration VERBOSE [U=0,P=109209] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime VERBOSE [U=0,P=109209] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts DEBUG [U=0,P=109209] addHomeStagingDir() Staging home directory (/root) at /usr/local/var/singularity/mnt/session/root DEBUG [U=0,P=109209] addHomeMount() Adding home directory mount [/usr/local/var/singularity/mnt/session/root:/root] to list using layer: overlay DEBUG [U=0,P=109209] addUserbindsMount() Adding /home/mrigter/github/mopo to mount list DEBUG [U=0,P=109209] addTmpMount() Checking for 'mount tmp' in configuration file VERBOSE [U=0,P=109209] addTmpMount() Default mount: /tmp:/tmp VERBOSE [U=0,P=109209] addTmpMount() Default mount: /var/tmp:/var/tmp DEBUG [U=0,P=109209] addScratchMount() Not mounting scratch directory: Not requested DEBUG [U=0,P=109209] addLibsMount() Checking for 'user bind control' in configuration file DEBUG [U=0,P=109209] addFilesMount() Checking for 'user bind control' in configuration file DEBUG [U=0,P=109209] addResolvConfMount() Adding /etc/resolv.conf to mount list VERBOSE [U=0,P=109209] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf DEBUG [U=0,P=109209] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request DEBUG [U=0,P=109209] create() Mount all DEBUG [U=0,P=109209] mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session DEBUG [U=0,P=109209] mountImage() Mounting loop device /dev/loop8 to /usr/local/var/singularity/mnt/session/rootfs of type squashfs DEBUG [U=0,P=109209] mountGeneric() Mounting overlay to /usr/local/var/singularity/mnt/session/final DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final DEBUG [U=0,P=109209] setPropagationMount() Set RPC mount propagation flag to SLAVE VERBOSE [U=0,P=109209] addIdentityMount() Not updating passwd/group files, running as root! DEBUG [U=0,P=109209] mountGeneric() Mounting /dev to /usr/local/var/singularity/mnt/session/final/dev DEBUG [U=0,P=109209] mountGeneric() Mounting /etc/localtime to /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC DEBUG [U=0,P=109209] mountGeneric() Mounting /etc/hosts to /usr/local/var/singularity/mnt/session/final/etc/hosts DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/etc/hosts DEBUG [U=0,P=109209] mountGeneric() Mounting /proc to /usr/local/var/singularity/mnt/session/final/proc DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/proc DEBUG [U=0,P=109209] mountGeneric() Mounting sysfs to /usr/local/var/singularity/mnt/session/final/sys DEBUG [U=0,P=109209] mountGeneric() Mounting /root to /usr/local/var/singularity/mnt/session/root DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/root DEBUG [U=0,P=109209] mountGeneric() Mounting /usr/local/var/singularity/mnt/session/root to /usr/local/var/singularity/mnt/session/final/root DEBUG [U=0,P=109209] mountGeneric() Mounting /tmp to /usr/local/var/singularity/mnt/session/final/tmp DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/tmp DEBUG [U=0,P=109209] mountGeneric() Mounting /var/tmp to /usr/local/var/singularity/mnt/session/final/var/tmp DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/var/tmp DEBUG [U=0,P=109209] mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/resolv.conf to /usr/local/var/singularity/mnt/session/final/etc/resolv.conf DEBUG [U=0,P=109209] mountGeneric() Mounting /home/mrigter/github/mopo to /usr/local/var/singularity/mnt/session/final/home/mopo DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/home/mopo DEBUG [U=0,P=109209] addCwdMount() Using /home/mrigter/github/mopo/singularity as current working directory VERBOSE [U=0,P=109209] addCwdMount() Not mounting CWD, /home/mrigter/github/mopo/singularity doesn't exist within container VERBOSE [U=0,P=109209] addCwdMount() Not mounting CWD, while getting /home/mrigter/github/mopo/singularity information: stat /usr/local/var/singularity/mnt/session/final/home/mrigter/github/mopo/singularity: no such file or directory DEBUG [U=0,P=109209] create() Chroot into /usr/local/var/singularity/mnt/session/final DEBUG [U=0,P=109232] Chroot() Hold reference to host / directory DEBUG [U=0,P=109232] Chroot() Called pivot_root on /usr/local/var/singularity/mnt/session/final DEBUG [U=0,P=109232] Chroot() Change current directory to host / directory DEBUG [U=0,P=109232] Chroot() Apply slave mount propagation for host / directory DEBUG [U=0,P=109232] Chroot() Called unmount(/, syscall.MNT_DETACH) DEBUG [U=0,P=109232] Chroot() Changing directory to / to avoid getpwd issues DEBUG [U=0,P=109209] create() Chdir into / to avoid errors VERBOSE [U=0,P=109226] wait_child() rpc server exited with status 0 DEBUG [U=0,P=109226] init() Set container privileges DEBUG [U=0,P=109226] apply_privileges() Effective capabilities: 0x0000003fffffffff DEBUG [U=0,P=109226] apply_privileges() Permitted capabilities: 0x0000003fffffffff DEBUG [U=0,P=109226] apply_privileges() Bounding capabilities: 0x0000003fffffffff DEBUG [U=0,P=109226] apply_privileges() Inheritable capabilities: 0x0000003fffffffff DEBUG [U=0,P=109226] apply_privileges() Ambient capabilities: 0x0000003fffffffff DEBUG [U=0,P=109226] apply_privileges() Set user ID to 0 DEBUG [U=0,P=109226] set_parent_death_signal() Set parent death signal to 9 DEBUG [U=0,P=109226] startup() singularity runtime engine selected VERBOSE [U=0,P=109226] startup() Execute stage 2 DEBUG [U=0,P=109226] StageTwo() Entering stage 2 DEBUG [U=0,P=109226] StartProcess() Setting umask in container to 0022 DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/10-docker2singularity.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/91-environment.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh DEBUG [U=0,P=109226] sylogBuiltin() Running action command exec FATAL [U=0,P=109226] StageTwo() permission denied DEBUG [U=0,P=109209] startContainer() stage 2 process reported an error, waiting status DEBUG [U=0,P=109209] Master() Child exited with exit status 255

To Reproduce Build container using the following definition file and execute using singularity exec as above:

Bootstrap: docker From: continuumio/miniconda3 Stage: devel

%setup

%files builder.py /home/ requirements.txt /home/ gpu-env.yml /home/ setup.py /home/

%environment export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/.mujoco/mujoco210/bin/ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/nvidia/lib64 export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so export MJLIB_PATH=/home/.mujoco/mujoco-2.1.1/lib/libmujoco.so.2.1.1 export MUJOCO_PY_MUJOCO_PATH=/home/.mujoco/mujoco210

%post apt-get update -q \ && DEBIAN_FRONTEND=noninteractive apt-get install -y \ build-essential \ wget \ curl \ git \ libgl1-mesa-dev \ libgl1-mesa-glx \ libglew-dev \ libopenmpi-dev \ libosmesa6-dev \ software-properties-common \ net-tools \ vim \ xpra \ nano \ xserver-xorg-dev

#Install Mujoco Binaries. 2.1.1 for dm_control and 2.1.0 for mujoco-py
mkdir /home/.mujoco
wget https://github.com/deepmind/mujoco/releases/download/2.1.1/mujoco-2.1.1-linux-x86_64.tar.gz -O mujoco.tar.gz
tar -xf mujoco.tar.gz -C /home/.mujoco
rm mujoco.tar.gz

wget https://mujoco.org/download/mujoco210-linux-x86_64.tar.gz -O mujoco.tar.gz
tar -xf mujoco.tar.gz -C /home/.mujoco
rm mujoco.tar.gz

# create environment
/opt/conda/bin/conda env create -f /home/gpu-env.yml
echo ". /opt/conda/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
echo "conda activate mopo" >> $SINGULARITY_ENVIRONMENT
. /opt/conda/etc/profile.d/conda.sh
conda activate mopo

#Create the mujoco environment variables
export MJLIB_PATH=/home/.mujoco/mujoco-2.1.1/lib/libmujoco.so.2.1.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/.mujoco/mujoco-2.1.1/bin/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/nvidia/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/.mujoco/mujoco210/bin
export MUJOCO_PY_MUJOCO_PATH=/home/.mujoco/mujoco210
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so

conda install -c conda-forge wandb
pip uninstall typing
pip install -r /home/requirements.txt

# install D4RL
git clone https://github.com/rail-berkeley/d4rl.git
rm d4rl/setup.py
cp /home/setup.py d4rl/
cd d4rl
pip install -e .
cd ..

#Install PatchELF for mujoco-py
curl -o /usr/local/bin/patchelf https://s3-us-west-2.amazonaws.com/openai-sci-artifacts/manual-builds/patchelf_0.9_amd64.elf
chmod +x /usr/local/bin/patchelf

# install libgcrypt11
wget http://archive.ubuntu.com/ubuntu/pool/main/g/glibc/multiarch-support_2.27-3ubuntu1.2_amd64.deb
dpkg -i multiarch-support_2.27-3ubuntu1.2_amd64.deb
wget https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+build/7110687/+files/libgcrypt11_1.5.4-2ubuntu1.1_amd64.deb
dpkg -i libgcrypt11_1.5.4-2ubuntu1.1_amd64.deb

rm libgcrypt11_1.5.4-2ubuntu1.1_amd64.deb
rm multiarch-support_2.27-3ubuntu1.2_amd64.deb

rm /opt/conda/envs/mopo/lib/python3.7/site-packages/mujoco_py/builder.py
cp /home/builder.py /opt/conda/envs/mopo/lib/python3.7/site-packages/mujoco_py/

python3 -c 'import mujoco_py'

%labels Author Marc Rigter Version v0.0.1

%help singularity run --nv -B .:/home/bind --network-args "portmap=8888:8888/tcp" mujoco.sif This container is a a demo for use with PyTorch and Cuda. Please use the flag --nv I'd suggest using the bind command to bind in the source code for building and using the bind command during executing to save data externally to the contaienr. The container also contains Jupyterlab for development

Expected behavior Expect the container to execute.

OS / Linux Distribution Ubuntu 20.04.3 LTS (Focal Fossa)

Installation Method Following these instructions: https://sylabs.io/guides/3.7/user-guide/quick_start.html

Additional context I've tried on two different machines using Singularity 3.7.3 and 3.9 and have the same issue in both cases.

marc-rigter commented 2 years ago

I've found that the issue for me was that the script that I was trying to run was not executable. Fixed by running chmod +x.

So not an issue with singularity, but the error message I was getting is not very informative.

dtrudg commented 2 years ago

Agreed we should likely check the if the target of exec is executable and have a nicer error if not.

dtrudg commented 2 years ago

Spent some time looking into it this morning, and am afraid this is not at all trivial so it'll be deferred. The error isn't particularly nice for users, but it is essentially the same as if you try run something non-exectuable in the host shell, so I'm going to consider a friendlier message an enhancement, not a bug:

11:37 AM $ /etc/hosts
bash: /etc/hosts: Permission denied

11:37 AM $ singularity exec ubuntu_latest.sif /etc/hosts
FATAL:   permission denied

The reason it's difficult to get something more friendly out is as follows....

It's easy to check if an explicit exec /etc/hosts is an executable file and report an error . However, once we start doing our own checks on the target of an exec we need to consider other cases. Consider singularity exec echo. This should generally run /bin/echo in the container. But it gets complicated due to the fact that shells implement their own version of echo, and if you run echo in a shell you will get that, not /bin/echo.

We use a shell 'action script' to drive the execution of processes in the container. For the singularity exec action this script runs exec "$@" - so the shell is going to try to resolve the executable that is passed as an argument. As far as the shell is concerned, permission denied is the correct error if you try to execute something to which you do not have +x permission.

It's actually quite difficult to find out exactly what a shell will exec... using only commands available in the shell. which is not a shell command, and has portability issues so cannot be used. POSIX sh has type but doesn't support the flags that bash does which allow you to find the path to a binary that is overridden by a shell built-in.

Our action script is run using an embedded shell interpreter, mvdan.cc/sh. This does have the type built-in that takes a -p flag. It'll then look on $PATH for an executable, and report the path to it if it exists. In theory some complex conditionals could be used around this, and checking paths. Howver, the type -p with mvdan.cc/sh doesn't match bash... and that project generally moves toward more bash over time so I'm concerned any complex conditionals we write might be fragile.

As a consequence of all the above, we'll add a note to the documentation about the reason for permission denied, but will defer enhancing the error message until it's clear there is time to do so, and in a future-proof manner. Apologies.