search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.54k stars 425 forks source link

unable to parse singularity.conf file: no such file #4774

Closed nick-youngblut closed 5 years ago

nick-youngblut commented 5 years ago

I'm running a VM with Ubuntu 18.04.3. With the fresh VM, I installed signularity 3.5.0 as described in the 3.5 user docs (except installed go 1.13 instead of 1.12 as incorrectly stated in the docs). I ran the quick start commands:

singularity pull library://sylabsed/examples/lolcow
singularity shell lolcow_latest.sif

...which generated the following error:

FATAL:   container creation failed: unable to parse singularity.conf file: open /usr/local/etc/singularity/singularity.conf: no such file or directory

...but my conf file exists (owned by root, and I'm running singularity as root):

-rw-r--r-- 1 root root 9.0K Nov 21 18:36 /usr/local/etc/singularity/singularity.conf
mj-harvey commented 5 years ago

I see this issue when taking a Centos 7.6 machine with 3.10.0-957.27.2.el7.x86_64 with a working EPEL singularity 3.2.1-1 to EPEL's singularity-3.4.2-1.1.el7.x86_64 

$ singularity -d run lowcow.sif
DEBUG   [U=1266233,P=850]  createConfDir()               /rds/general/user/slacalle/home/.singularity already exists. Not creating.
DEBUG   [U=1266233,P=850]  getCacheBasedir()             environment variable SINGULARITY_CACHEDIR not set, using default image cache
DEBUG   [U=1266233,P=850]  updateCacheSubdir()           Caching directory set to /rds/general/user/slacalle/home/.singularity/cache/library
DEBUG   [U=1266233,P=850]  updateCacheSubdir()           Caching directory set to /rds/general/user/slacalle/home/.singularity/cache/oci-tmp
DEBUG   [U=1266233,P=850]  updateCacheSubdir()           Caching directory set to /rds/general/user/slacalle/home/.singularity/cache/oci
DEBUG   [U=1266233,P=850]  updateCacheSubdir()           Caching directory set to /rds/general/user/slacalle/home/.singularity/cache/net
DEBUG   [U=1266233,P=850]  updateCacheSubdir()           Caching directory set to /rds/general/user/slacalle/home/.singularity/cache/shub
DEBUG   [U=1266233,P=850]  updateCacheSubdir()           Caching directory set to /rds/general/user/slacalle/home/.singularity/cache/oras
DEBUG   [U=1266233,P=850]  execStarter()                 Use starter binary /usr/libexec/singularity/bin/starter-suid
DEBUG   [U=1266233,P=850]  execStarter()                 Checking for encrypted system partition
DEBUG   [U=1266233,P=850]  Init()                        Image format detection
DEBUG   [U=1266233,P=850]  Init()                        Check for sandbox image format
DEBUG   [U=1266233,P=850]  Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=1266233,P=850]  Init()                        Check for sif image format
DEBUG   [U=1266233,P=850]  Init()                        sif image format detected
VERBOSE [U=1266233,P=850]  SetContainerEnv()             HOME = /rds/general/user/slacalle/home
VERBOSE [U=0,P=850]        print()                       Set messagelevel to: 5
VERBOSE [U=0,P=850]        init()                        Starter initialization
DEBUG   [U=0,P=850]        get_pipe_exec_fd()            PIPE_EXEC_FD value: 9
VERBOSE [U=0,P=850]        is_suid()                     Check if we are running as setuid
VERBOSE [U=0,P=850]        priv_drop()                   Drop root privileges
DEBUG   [U=1266233,P=850]  init()                        Read engine configuration
DEBUG   [U=1266233,P=850]  init()                        Wait completion of stage1
VERBOSE [U=1266233,P=861]  priv_drop()                   Drop root privileges permanently
DEBUG   [U=1266233,P=861]  set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1266233,P=861]  init()                        Spawn stage 1
DEBUG   [U=1266233,P=861]  startup()                     singularity runtime engine selected
VERBOSE [U=1266233,P=861]  startup()                     Execute stage 1
DEBUG   [U=1266233,P=861]  StageOne()                    Entering stage 1
DEBUG   [U=1266233,P=861]  Init()                        Image format detection
DEBUG   [U=1266233,P=861]  Init()                        Check for sandbox image format
DEBUG   [U=1266233,P=861]  Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=1266233,P=861]  Init()                        Check for sif image format
DEBUG   [U=1266233,P=861]  Init()                        sif image format detected
VERBOSE [U=1266233,P=850]  wait_child()                  stage 1 exited with status 0
DEBUG   [U=1266233,P=850]  cleanup_fd()                  Close file descriptor 4
DEBUG   [U=1266233,P=850]  cleanup_fd()                  Close file descriptor 5
DEBUG   [U=1266233,P=850]  cleanup_fd()                  Close file descriptor 7
DEBUG   [U=1266233,P=850]  cleanup_fd()                  Close file descriptor 8
DEBUG   [U=1266233,P=850]  init()                        Set child signal mask
DEBUG   [U=1266233,P=850]  init()                        Create socketpair for master communication channel
DEBUG   [U=1266233,P=850]  init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1266233,P=850]  priv_escalate()               Get root privileges
VERBOSE [U=0,P=850]        priv_escalate()               Change filesystem uid to 1266233
VERBOSE [U=0,P=850]        init()                        Spawn master process
DEBUG   [U=0,P=868]        set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=868]        create_namespace()            Create mount namespace
VERBOSE [U=0,P=850]        enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=850]        enter_namespace()             Opening namespace file ns/mnt
DEBUG   [U=0,P=850]        setns()                       Using syscall() wrapped __NR_setns
VERBOSE [U=0,P=868]        create_namespace()            Create mount namespace
DEBUG   [U=0,P=869]        set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=869]        init()                        Spawn RPC server
DEBUG   [U=1266233,P=850]  startup()                     singularity runtime engine selected
VERBOSE [U=1266233,P=850]  startup()                     Execute master process
FATAL   [U=1266233,P=850]  Master()                      container creation failed: unable to parse singularity.conf file: open /etc/singularity/singularity.conf: no such file or directory
DEBUG   [U=0,P=869]        startup()                     singularity runtime engine selected
VERBOSE [U=0,P=869]        startup()                     Serve RPC requests
cclerget commented 5 years ago

@nick-youngblut Could you provide step in order to reproduce it ? Is it a vagrant/virtualbox VM ? and is there any particular setup done before running singularity ?

Between another user on slack reported a similar issue with epel package on centos7.7

mj-harvey commented 5 years ago

Further -- have no ~/.singularity customisation, and the /etc/singularity/singularity.conf comes from the EPEL package (with different contents to that from the 3.2.1 package)

Issue also occurs if singularity is run as root

I have found a Centos 7.6 machine with the latest epel singularity, which does work. The point of divergence in the debug trace is:


VERBOSE [U=0,P=4887]       startup()                     Serve RPC requests
DEBUG   [U=97933,P=4143]   checkOverlay()                Overlay seems supported and allowed by kernel
DEBUG   [U=97933,P=4143]   setupSessionLayout()          Attempting to use overlayfs (enable overlay = try)
DEBUG   [U=97933,P=4143]   setupOverlayLayout()          Creating overlay SESSIONDIR layout
'''
cclerget commented 5 years ago

@mj-harvey And on a Centos 7.7 machine with the latest epel, does it work too ? Trying to determine if that come from a recent change in kernel like a backported patch.

nick-youngblut commented 5 years ago

The VM was created with a lightweight container framework that is very similar to lxc.

My setup was simply as described in the quick start:

# from a brand new VM
$ sudo apt-get update && sudo apt-get install -y \
    build-essential \
    libssl-dev \
    uuid-dev \
    libgpgme11-dev \
    squashfs-tools \
    libseccomp-dev \
    wget \
    pkg-config \
    git \
    cryptsetup

$ export VERSION=1.13 OS=linux ARCH=amd64 && \
  wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \ package
  sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
  rm go$VERSION.$OS-$ARCH.tar.gz    # Deletes the ``tar`` file

$ echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.bashrc && \
  source ~/.bashrc

$ export VERSION=3.5.0 &&  \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
    tar -xzf singularity-${VERSION}.tar.gz && \
    cd singularity

$ ./mconfig && \
    make -C builddir && \
    sudo make -C builddir install

$ singularity pull library://sylabsed/examples/lolcow

$ singularity shell lolcow_latest.sif
cclerget commented 5 years ago

@nick-youngblut What is the lightweight container framework you mentioned ? Also have you try with the recent version of Go instead : export VERSION=1.13.4 ?

mj-harvey commented 5 years ago

@cclerget the latest epel release does not work on the centos 7.7 machine. that is fully up to date, but it's not a fresh install.

I'm not able to try the older 3.2.1 epel release on a centos 7.7 system at the moment, but can in a day or so.

nick-youngblut commented 5 years ago

The lightweight container framework was created by our admin. Here's the link: http://people.tuebingen.mpg.de/maan/micoforia/

The admin can make the source code available. Here's the main details:

cclerget commented 5 years ago

@mj-harvey I'm not able to reproduce the issue on Centos 7.7, with a fresh vagrant box bento/centos7.7:

@mj-harvey is SELinux enabled (and enforced) on Centos7.7 ?

@nick-youngblut Thanks for the information, the author could help us to determine what's wrong, the issue is really weird because singularity.conf is open/read multiple time during the workflow but the last open fail with no such file or directory which means something mess up with mount namespaces because the master process opening singularity.conf joined the first "shared" mount namespace created by the final container process, and it looks like the master process joined a mount namespace where /usr/local/etc/singularity/singularity.conf is not present like joining a mount namespace where a pivot_root already occured or a mount namespace residing in another root filesystem, so it make me think that the master process in reality joined the initial host mount namespace where /usr/local/etc/singularity/singularity.conf is not present. Here few questions :

mj-harvey commented 5 years ago

@cclerget selinux is either disabled or in permissive mode on my test machines. I've just tried 3.5.0-1 from epel-testing on the problematic machines, and see the same behaviour.

For now I'm going to have to revert to 3.2.1, but what else can I do to debug this?

Additional info from nonworking machine:

# mount | grep -v "run/user"
/dev/sda on / type xfs (rw,relatime,inode64,noquota)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=99006112k,nr_inodes=24751528,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=346)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
rds on /rdsgpfs type gpfs (rw,relatime)
mj-harvey commented 5 years ago

@cclerget I just tested 3.3.0-1 from epel, and note that this also exhibits the same broken "file not found" behaviour. We'd missed out on testing that release, planning to go from 3.2.1 to 3.4.2 to fit in with our refresh cycle.

cclerget commented 5 years ago

@mj-harvey Thanks for the information, I remembered an issue about a weird behavior like that (https://github.com/sylabs/singularity/issues/2000) and after verification the line fixing the issue https://github.com/sylabs/singularity/blob/1482fab04dcab6ba4d0cbe2098aa82310aff0ef9/cmd/starter/c/starter.c#L1025 has been removed since 3.3 release so that could explain the issue here.

mj-harvey commented 5 years ago

starter.c has changed quite a bit between 3.2.1 and 3.4.2, but I stuffed a sched_yield() in immediately beforeuser_namespace_init(&sconfig->container.namespace) at starter.c:1076 . It didn't make a difference. debug trace:

...
DEBUG   [U=0,P=24781]      cleanup_fd()                  Close file descriptor 4
DEBUG   [U=0,P=24781]      init()                        Set child signal mask
DEBUG   [U=0,P=24781]      init()                        Create socketpair for master communication channel
DEBUG   [U=0,P=24781]      init()                        Create RPC socketpair for communication between stage 2 and RPC server
DEBUG   [U=0,P=24781]      init()                        MJH sched_yield
DEBUG   [U=0,P=24781]      init()                        MJH sched_yield
VERBOSE [U=0,P=24781]      priv_escalate()               Get root privileges
VERBOSE [U=0,P=24781]      priv_escalate()               Change filesystem uid to 0
VERBOSE [U=0,P=24781]      init()                        Spawn master process
DEBUG   [U=0,P=24799]      set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=24799]      create_namespace()            Create mount namespace
VERBOSE [U=0,P=24781]      enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=24781]      enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=24799]      create_namespace()            Create mount namespace
DEBUG   [U=0,P=24800]      set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=24800]      init()                        Spawn RPC server
DEBUG   [U=0,P=24781]      startup()                     singularity runtime engine selected
VERBOSE [U=0,P=24781]      startup()                     Execute master process
DEBUG   [U=0,P=24800]      startup()                     singularity runtime engine selected
VERBOSE [U=0,P=24800]      startup()                     Serve RPC requests
FATAL   [U=0,P=24781]      Master()                      container creation failed: unable to parse singularity.conf file: open /etc/singularity/singularity.conf: no such file or directory
mj-harvey commented 5 years ago

Looking at the source, I see there are several locatation that might give rise to that error. Just in case it's not obvious to you, it's coming from internal/pkg/runtime/engines/singularity/create_linux.go:30 (v3.4.2 branch)

cclerget commented 5 years ago

@mj-harvey Could you try with #4785 ?

mj-harvey commented 5 years ago

The issue was that the initrd for the systems exhibiting the problem used the idiom

cd /a
pivot_root /a /sysroot
exec chroot /a /sbin/init

But because the initrd / fs is rootfs, the pivot_root was failing. The result is that the kernel's idea of the filesystem root remaind unchanged, even though userspace's (inherited down through init) had changed.

From version 3.3, singularity's use of namespaces changed, with the result that creating a new fs namespace left singularity inside the rootfs /.

The fix for this was to change pivot_root+chroot to switch_root(8)

Thanks @cclerget for solving this!

nick-youngblut commented 5 years ago

Here's the response from our admin:

Is the framework use chroot or pivot_root when spawning the container ? because chroot is easily breakable

It first changes the cwd to the container root, then calls chroot("."), then execve() to exec the container init.

Is there seccomp filters set by the framework making unshare syscall a sort of noop or is the framework allow nested containers ?

No seccomp filters. Not sure about nested containers.

My response to your last question:

Is an older version of Singularity like 3.2.1 work within a VM with this framework ?

I tried to install Singularity 3.2.1 and 3.3.0, and ran into installation problems with both (issues with go during the build). Maybe that is the result of not uninstalling singularity 3.5 first (is that even possible?)

cclerget commented 5 years ago

It first changes the cwd to the container root, then calls chroot("."), then execve() to exec the container init.

@nick-youngblut Thanks for the information, I think you will need to patch the framework to use pivot_root (or mount with MS_MOVE) instead because it won't be fixed in Singularity even if we could but that's not going to serve you by doing that, what causing the issue with the framework is a container escape, even by using user namespace someone can escape the container's chroot by executing in the container :

unshare -U -m sleep 100 &
nsenter -t $! -U -m --preserve-credentials ls -la /

Last command would list the root directory of the host filesystem not the container one.

yanpd01 commented 2 years ago

So is there any way to fix this, I now have the same problem on both of my servers, how can I fix it. One of the servers is ubuntu 20 and the other is ubuntu 22.

yanpd01 commented 2 years ago

So is there any way to fix this, I now have the same problem on both of my servers, how can I fix it. One of the servers is ubuntu 20 and the other is ubuntu 22.

Sorry to disturb everyone. I have found the answer.

When I compiled and installed singularity, I used ./mconfig --prefix=~/usr/apps/singularity to specify the installation path, which caused my problem.

When I use absolute path ./mconfig --prefix=/home/user_name/usr/apps/singularity it works fine.