search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

exec leads to Failed to mount image in (read only): Invalid argument #471

Closed yarikoptic closed 6 years ago

yarikoptic commented 7 years ago

Hi - me is back with fresh whinings but possibly using not so fresh but released singularity 2.2 (from Debian), definition file is quite simple: http://www.onerussian.com/tmp/dbic-bids-env.def

I wonder if it somehow relates to relatively large size of the image (reserved for 3GB, used slightly over 2GB):

$> du -scm dbic-bids-env.img 
2526    dbic-bids-env.img
2526    total

$> du -scm --apparent-size dbic-bids-env.img
3049    dbic-bids-env.img
3049    total

previous attempt was done with smaller image (2GB) and fewer packages installed -- everything worked out fine, but then added more packages, ran out of space, increased size, and kaboom. now tried to reproduce with smaller size -- still kaboom :-/

creation/bootstrap steps:

$> sudo singularity create -s 3048 dbic-bids-env.img && sudo singularity bootstrap dbic-bids-env.img dbic-bids-env.def
...
Done.
sudo singularity bootstrap dbic-bids-env.img dbic-bids-env.def  102.07s user 18.41s system 30% cpu 6:34.68 total

and then attempt to exec:

$> singularity exec dbic-bids-env.img /bin/bash
ERROR  : Failed to mount image in (read only): Invalid argument
ABORT  : Retval = 255

kernel (4.8.0-2-amd64 #1 SMP Debian 4.8.11-1 (2016-12-02) x86_64 GNU/Linux) messages include:

Jan 28 00:33:09 hopa kernel: [53133.297965] EXT4-fs (loop7): mounting ext3 file system using the ext4 subsystem
Jan 28 00:33:09 hopa kernel: [53133.298008] EXT4-fs (loop7): INFO: recovery required on readonly filesystem
Jan 28 00:33:09 hopa kernel: [53133.298009] EXT4-fs (loop7): write access unavailable, cannot proceed

what could be a culprit? can you reproduce it or just my luck/OS/version/filesystem? (now tried a few times)

P.S. why generated image is "executable"?:

$> ls -ld ./dbic-bids-env.img
-rwxr-xr-x 1 root yoh 3196059679 Jan 28 00:23 ./dbic-bids-env.img*
vsoch commented 7 years ago

I think it's because you installed that eatmydata bit? I just bootstrapped the image, and I reproduced your error, and in writable mode it DID work. I'm going to try making the image without that bit and see if it works.

vsoch commented 7 years ago

ok so I just tried the following:

I think it worked! but would need testing.. I just shelled in and did useless things. Here is the spec file, if it's useful:

# Copyright (c) 2015-2016, Gregory M. Kurtzer. All rights reserved.
# Changes for NeuroDebian setup are Copyright (c) 2017 Yaroslav Halchenko.
#
# "Singularity" Copyright (c) 2016, The Regents of the University of California,
# through Lawrence Berkeley National Laboratory (subject to receipt of any
# required approvals from the U.S. Dept. of Energy).  All rights reserved.

BootStrap: docker
From: debian:latest

%runscript
    echo "This is what happens when you run the container..."

%post
    echo "Hello from inside the container"
    apt-get update
    apt-get -y install vim eatmydata wget
    wget -q -O/tmp/nd-configurerepo https://raw.githubusercontent.com/neurodebian/neurodebian/4d26c8f30433145009aa3f74516da12f560a5a13/tools/nd-configurerepo
    bash /tmp/nd-configurerepo
    eatmydata apt-get -y install datalad python-nipype virtualenv dcm2niix

I can't believe they sneak in a "package use survey." Really?!

yarikoptic commented 7 years ago

not sure yet who or how but something seems to hold on to those .img files -- I have generated some to test in /tmp and quickly ran out of space there although there is no large files there. So some (or all?) image files remained opened by some process... will check tomorrow in detail but I even see:

cron      15800                    root    5u      REG              253,0 805306399    8850010 /home/yoh/deb/gits/pkg-exppsy/singularity-container/examples/debian-test.img (deleted)

not yet sure what cron specifically needs to do having that file open... ?

vsoch commented 7 years ago

Do you mean the cache? Did you look at $HOME/.singularity ?

vsoch commented 7 years ago

I'm not sure what you are referencing by "hold on to," but based on saying "run out of space" I hope we don't have a bug somewhere.. Docker layers ate my computer last week, and more than a kernel possibly went into panic, lol.

yarikoptic commented 7 years ago

"hold on" == e.g. "open for writing and never close", so that FID/space remains reserved even if you remove that file from the filesystem. You can still access that file through its fid linked to the process... That particular cron job seems to possibly be coming from within the container or from when it was generated/bootstrapped:

# cat /proc/15800/environ 
SUDO_GID=47522MAIL=/var/mail/rootUSER=rootNEW_HOME=SINGULARITY_localstatedir=/varSHLVL=1OLDPWD=/HOME=/rootPS1=${debian_chroot:+($debian_chroot)}\u@\h:\w\$ COLORTERM ...
# ls -l /proc/15800/fd/5
lrwx------ 1 root root 64 Jan 30 12:14 /proc/15800/fd/5 -> /home/yoh/deb/gits/pkg-exppsy/singularity-container/examples/debian-test.img (deleted)

but I failed to determine this way what exactly ate my /tmp space :-/ would need to reboot I guess

bbockelm commented 7 years ago

If you can get the PID from the output of ps faux, you can do the following:

lsof -p $PID

That'll show the files which are being held open. Alternately, if you can't figure out which process has the open files, you can do:

lsof | grep /tmp

PS - the containers are executable because the image files are literally executable. If you try to execute it, it'll run the contents of %runscript.

yarikoptic commented 7 years ago

that is how I got ahold of that rogue cron -- via lsof and grep. Didn't have yet time to reboot to start anew and shouldn't grow my /tmp any further ATM ;) I have also explored all /proc/*/fd/* to see if any point to some deleted file under /tmp -- nothing... and I am still ~2.5G short of space in the /tmp/ ... heh

PS -- thanks for the executable bit explanation

yarikoptic commented 7 years ago

heh -- was trying to chase it down but failed and now really need to reboot, but here is another piece of a "proof":

$> singularity exec dbic-bids-env-part.img /bin/bash                                                                                                                                                                                 
ERROR  : Failed to mount image in (read only): Invalid argument
ABORT  : Retval = 255

$> df . -h
Filesystem      Size  Used Avail Use% Mounted on
tmp             6.0G  4.6G  1.4G  78% /tmp

$> du -scm dbic-bids-env-part.img 
1119    dbic-bids-env-part.img
1119    total

$> du -scm --apparent-size dbic-bids-env-part.img
1401    dbic-bids-env-part.img
1401    total

$> sudo fuser dbic-bids-env-part.img
$> singularity exec dbic-bids-env-part.img /bin/bash
ERROR  : Failed to mount image in (read only): Invalid argument
ABORT  : Retval = 255

$> df . -h                                          
Filesystem      Size  Used Avail Use% Mounted on
tmp             6.0G  4.6G  1.4G  78% /tmp

$> sudo rm dbic-bids-env-part.img 

$> df . -h                                          
Filesystem      Size  Used Avail Use% Mounted on
tmp             6.0G  4.6G  1.4G  78% /tmp

as you can see -- even after "removed" -- space is still claimed for. And that is the case ONLY for those images where 'exec' failed. I haven't checked yet if calling exec on such an image is the requirement to get space "stuck". But I guess that some kind of a mount it is trying to do is failing, but not releasing some resources.

bbockelm commented 7 years ago

Oh! Can you look at /proc/$PID/mountinfo and see if there are lingering mounts? The kernel itself might hang onto the mount, independent of a process having it open.

Particularly - is there any suspicious-looking loopback mounts that might be holding on to this mount point?

yarikoptic commented 7 years ago

@bbockelm of which $PID? or of all?

yarikoptic commented 7 years ago

I guess it could be those:

$> grep loop /proc/*/mountinfo      
/proc/12155/mountinfo:382 269 7:2 / / rw,nosuid,relatime - ext3 /dev/loop2 rw,errors=remount-ro,data=ordered
/proc/13233/mountinfo:382 269 7:2 / / rw,nosuid,relatime - ext3 /dev/loop2 rw,errors=remount-ro,data=ordered
/proc/13307/mountinfo:382 269 7:2 / / rw,nosuid,relatime - ext3 /dev/loop2 rw,errors=remount-ro,data=ordered
2 10034.....................................:Tue 31 Jan 2017 12:28:34 AM EST:.
hopa:~
$> ps auxw | grep 12155
root     12155  0.0  0.0  25904  2236 ?        Ss   00:26   0:00 /usr/sbin/cron
yoh      14419  0.0  0.0  12720   972 pts/15   S+   00:28   0:00 grep --color=auto -d skip 12155
2 10035.....................................:Tue 31 Jan 2017 12:28:42 AM EST:.
hopa:~
$> ps auxw | grep 13233
systemd+ 13233  0.0  0.0  53256  3268 ?        Ss   00:26   0:00 /usr/sbin/exim4 -bd -q30m
yoh      14457  0.0  0.0  12720   964 pts/15   S+   00:28   0:00 grep --color=auto -d skip 13233
2 10036.....................................:Tue 31 Jan 2017 12:28:51 AM EST:.
hopa:~
$> ps auxw | grep 13307
message+ 13307  0.0  0.0  42124  2192 ?        Ss   00:26   0:00 /usr/bin/dbus-daemon --system
yoh      14494  0.0  0.0  12720   976 pts/15   S+   00:28   0:00 grep --color=auto -d skip 13307

After I killed all 3 of those space was recovered (this time did in a temporary tmpfs mount ;) )! so indeed I guess it is some processes started during installation within the 'container'.

yarikoptic commented 7 years ago

few more bits: details and cwd's for those commands which hand it up:

$> grep -l loop /proc/*/mountinfo | sed -e 's,/proc/\(.*\)/.*,\1,g' | while read pid; do ps -up $pid; sudo ls -ld /proc/$pid/cwd; done                                                                                     
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     26579  0.0  0.0  25904  2240 ?        Ss   00:34   0:00 /usr/sbin/cron
lrwxrwxrwx 1 root root 0 Jan 31 00:36 /proc/26579/cwd -> /var/lib/singularity/mnt/final/var/spool/cron
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
systemd+ 27675  0.0  0.0  53248  3212 ?        Ss   00:34   0:00 /usr/sbin/exim4 -bd -q30m
lrwxrwxrwx 1 root root 0 Jan 31 00:37 /proc/27675/cwd -> /var/lib/singularity/mnt/final/var/spool/exim4
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
message+ 27749  0.0  0.0  42124  2284 ?        Ss   00:34   0:00 /usr/bin/dbus-daemon --system
lrwxrwxrwx 1 root root 0 Jan 31 00:37 /proc/27749/cwd -> /var/lib/singularity/mnt/final

and if I kill those -- I can exec into the image! so I wonder if singularity should aggressively sanitize some rogue kids processes somehow upon exiting the environment?

dominik-handler commented 7 years ago

I just now had exactly the same problem. Could resolve the whole thing by once executing: sudo singularity shell -w img this somehow cleaned up the service processes and makes the container executable.

rajaseelan commented 7 years ago

Hi, I am having the same problem.

I did a df -hl and then realised this:

[root@server images]# df -hl
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   12G   39G  23% /
devtmpfs                  63G     0   63G   0% /dev
tmpfs                     63G   88K   63G   1% /dev/shm
tmpfs                     63G   50M   63G   1% /run
tmpfs                     63G     0   63G   0% /sys/fs/cgroup
/dev/sda2                494M  159M  336M  33% /boot
/dev/sda1                200M  9.5M  191M   5% /boot/efi
/dev/mapper/centos-home  106G   53M  106G   1% /home
tmpfs                     13G   12K   13G   1% /run/user/0
/dev/loop0               976M  329M  597M  36% /run/media/root/3a6fdca0-487d-4f5e-8829-2ded842e5ca2
/dev/loop1               976M  404M  522M  44% /run/media/root/f15c576a-c463-4f48-88c6-c4dd7651a321
/dev/loop2               976M  329M  597M  36% /run/media/root/923034dc-90a0-47aa-805a-4d9375ea4192
/dev/loop3               976M  329M  597M  36% /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f27
/dev/loop4               976M  1.3M  924M   1% /run/media/root/51253483-e074-4068-8975-81045d12c30e
/dev/loop5               976M  329M  597M  36% /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f271
/dev/loop6               976M  329M  597M  36% /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f272
/dev/loop7               976M  404M  522M  44% /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d071662
/dev/loop8               976M  404M  522M  44% /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d0716621
/dev/loop9               976M  404M  522M  44% /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d0716622

So i did the following:

[root@server images]# umount -v /dev/loop*
umount: /run/media/root/3a6fdca0-487d-4f5e-8829-2ded842e5ca2 (/dev/loop0) unmounted
umount: /run/media/root/f15c576a-c463-4f48-88c6-c4dd7651a321 (/dev/loop1) unmounted
umount: /dev/loop10: not mounted
umount: /run/media/root/923034dc-90a0-47aa-805a-4d9375ea4192 (/dev/loop2) unmounted
umount: /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f27 (/dev/loop3) unmounted
umount: /run/media/root/51253483-e074-4068-8975-81045d12c30e (/dev/loop4) unmounted
umount: /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f271 (/dev/loop5) unmounted
umount: /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f272 (/dev/loop6) unmounted
umount: /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d071662 (/dev/loop7) unmounted
umount: /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d0716621 (/dev/loop8) unmounted
umount: /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d0716622 (/dev/loop9) unmounted
umount: /dev/loop-control: not mounted

I can now do a singularity shell centos.img

I did a mount command, it still kept the loop mounts after I exited.

/opt/images/ansible.img on /run/media/root/6b48edd8-4f74-4ccb-9bc7-1b175d071662 type ext3 (ro,nosuid,nodev,relatime,errors=remount-ro,data=ordered,uhelper=udisks2)
/opt/images/centos.img on /run/media/root/166c2959-20f5-4d7f-96b3-ea14afea4f27 type ext3 (ro,nosuid,nodev,relatime,errors=remount-ro,data=ordered,uhelper=udisks2)

CentOS 7 with the latest updates.

Attached is the def files

ansible.def.txt centos.def.txt

gmkurtzer commented 7 years ago

@yarikoptic and @rajaseelan: The only time you should see that is if there are rogue processes still running within that mount namespace (e.g. did a process fork and run in the background as a daemon?). Perhaps this was discussed above already?

What version of Singularity did you find this on? Can you test the "development" github branch?

Thanks!

yarikoptic commented 7 years ago

@gmkurtzer

The only time you should see that is if there are rogue processes still running within that mount namespace (e.g. did a process fork and run in the background as a daemon?)

yeap -- but that is a normal state of things whenever you install any daemon, right? so somehow bootstrap should not be bringing those beasts up at all (not sure if possible to prevent it in a scalable fashion) or trace and then kill those child ones

What version of Singularity did you find this on? Can you test the "development" github branch?

  • released singularity 2.2
  • I will try to try development later on today
gmkurtzer commented 7 years ago

Oh, you have a good point. On RedHat, an installation of an RPM never (that I know of) would actually start the daemon. Does it on Debian based OSs?

It might be possible to bootstrap within a PID namespace, and then do a killall within that namespace actually... I will check into this.

vjkumar1 commented 7 years ago

@yarikoptic & @gmkurtzer -- Were you able to digup anything more on this? I ran into the same issue while tinkering with some test jobs in the container. Don't have a set of steps to reproduce it on demand yet but this raises a bit of concern on our go forward plans. Thanks,

-Vijay

gmkurtzer commented 7 years ago

Hi @vjkumar1 If you are testing anything but the latest development branch (as of 2 hours ago), this might be wrong. It seems that /etc/mtab was being updated when it shouldn't have been, and not being cleaned. In practice, the mounts were not persistent, but commands like df and mount would give false positives.

Can you retest with latest development branch?

Thanks!

yarikoptic commented 7 years ago

Ha ha... We should coin the term "continuous(ly) bleeding development" :-)

vsoch commented 7 years ago

Gotta be true to continuous deployment @yarikoptic ! :octopus:

yarikoptic commented 7 years ago

"continuous deployment" is old news and doesn't depict the bleeding aspect ;)
note that "continuous deployment" term is primarily used for online services where users have no chance by "to upgrade", so introduced issues are discovered more rapidly and everyone suffers either they want it or not. With tools situation is a bit different -- usually it takes some dedication and sacrifice to step on the bleeding edge making a fully conscious decision.

dtrudg commented 6 years ago

Hi @yarikoptic - this issue was reported against Singularity 2.2. Do you experience any issue running the latest 2.4 series? If not I will close the ticket. Thanks!

yarikoptic commented 6 years ago

I don't think i have ran into similar situations recently so let's close -- but I might come back and reopen if I do hit it again

Bennio commented 5 years ago

before running your program make sure to exit from singularity shell

abhinavdrs commented 5 years ago

Facing a similar isssue on Singularity version 3.2.1-1.el7: Command singularity run --contain -B /tmp/data:/data,/tmp/.cache:/.cache --nv ./strudl.sif --port 8000 Error: FATAL: container creation failed: mount /proc/self/fd/12->/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/12: failed to mount squashfs filesystem: invalid argument.

OS: Machine details:

NAME="Red Hat Enterprise Linux Server" VERSION="7.4 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.4" PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)"

jscook2345 commented 5 years ago

@abhinavdrs

Can you open a new issue with your information provided so we can help you there?

Can you also include the output of lsmod for the system you're running on? Thanks!

abhinavdrs commented 5 years ago

Hello @jscook2345, thanks. I created a new issue earlier today. https://groups.google.com/a/lbl.gov/forum/?utm_medium=email&utm_source=footer#!msg/singularity/ATO7f_hMO0w/B2GfSPfFCgAJ

I will attach the lsmod output there.