User: "no space left on device" but not at Root

[AhmedBahrani]

Version of Singularity:3.6.0-rc.4

What version of Singularity are you using? Run: $ singularity version: 3.6.0

singularity build -F aa.sif aa.def singularity run -e -C -H :/opt/data ab_s.sif bash /opt/s/aa.sh

Expected behavior

I am running a bash script build in a singularity container as shown above and I am expecting to get several nifti images in the output folder, and that happened properly when I ran the command "singularity run" as a ROOT.

So, I am expecting: running the container as a "USER" should give the exact same results as a "ROOT" login.

Actual behavior

However, running the script as USER stops at a certain point and shows the following:

The error commands:

ncvarput: ncid 4: No space left on device mivarput1: MINC package entry point nu_evaluate: crashed while running minccalc (termination status=768) nu_correct: crashed while running nu_evaluate (termination status=768)

What actually happened? Why was it incorrect? Using the USER login gives an error.

Steps to reproduce this behavior

How can others reproduce this issue/problem? Using my container and the MRI nifti images

What OS/distro are you running

Red Hat Enterprise Linux Workstation 7.9

$ cat /etc/os-release

How did you install Singularity

YUM Write here how you installed Singularity. Eg. RPM, source.

[dtrudg]

We need more information to diagnose this. Does the script write any files as the non-root user? How full is the filesystem that it is writing to, and when root is used how much space is used/how full does it get?

If you are close to filling a filesystem you may see differences between root and non-root. Linux filesystems generally, by default, reserve a percentage of space that can only be used by the root user - so a non-root user will see 'no space left' before root does.

[AhmedBahrani]

Thanks Dctrud,

The script was written completely as a non-root user but the "singularity build" was run as a root user. When I built the container the hard drive space was around 100 GB and the root space was ~70 GB. During the container build execution, the "tmpfs" file used about 30 GB. The container size is 10.6 GB. Currently, the available size of the hard drive is 77 GB and the root is 62 GB.

The other thing that I have noticed that the freeSurfer command "nu_correct" is firstly fail and corrupted everything as a non-root user. Please, see the error message below. However, "nu_correct" execution did not fail as a root user.

CV of field change: 0.00113578 evaluate_field: posixio.c:210: px_pgin: Assertion `posp == ((off_t)(-1)) || posp == lseek(nciop->fd, 0, 1)' failed. nu_evaluate: crashed while running evaluate_field (termination status=134) nu_correct: crashed while running nu_evaluate (termination status=65280)

The other thing that I also noticed is that the starter-suid is shaded by red color and the font color also red even though the permission line is correct as shown below:

-rwxr-xr-x. 1 root root 22134128 Jun 2 14:21 starter -rwsr-xr-x. 1 root root 21786504 Jun 2 14:21 starter-suid

The "starter" is not highlighted and the font color is green.

[carterpeel]

Hello,

This is a templated response that is being sent out to all open issues. We are working hard on 'rebuilding' the Singularity community, and a major task on the agenda is finding out what issues are still outstanding.

Please consider the following:

1. Is this issue a duplicate, or has it been fixed/implemented since being added?
2. Is the issue still relevant to the current state of Singularity's functionality?
3. Would you like to continue discussing this issue or feature request?

Thanks, Carter

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in over 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because no response was provided within 7 days.