"can't remount" under "cleaned up" /home and /tmp in -c mode

[yarikoptic]

Version of Singularity:

Initially ran into a failure with 2.6.1 where failing behavior varies (might be related to #3530)

```shell /tmp/d > mkdir -p {~,/tmp}/d; touch {~,/tmp}/d/file; for d in /etc /etc/apt ~ ~/d /tmp /tmp/d; do echo $d ; builtin cd $d; singularity exec -c -B $d /tmp/out1/containers/images/bids/bids-mriqc--0.15.0.sing ls -l $d >/dev/null; echo "exited with $?"; done /etc exited with 0 /etc/apt exited with 0 /home/yoh WARNING: Not mounting requested bind point (already mounted in container): /home/yoh exited with 0 /home/yoh/d exited with 0 /tmp exited with 0 /tmp/d ls: cannot access '/tmp/d': No such file or directory exited with 2 ```

and then replicated with 3.1.1 (debian system both times)

Expected behavior

leading subdirectories will be provided within cleaned up environment, similar to how it is done to any other directory (overlays?)

Actual behavior

Just fails to bind mount a directory under /home or /tmp whenever running with -c flag:

/tmp/d > mkdir -p {~,/tmp}/d; touch {~,/tmp}/d/file; for d in /etc /etc/apt /data/famface/ ~ ~/d /tmp /tmp/d; do echo $d ; builtin cd $d; singularity exec -c -B $d /tmp/out1/containers/images/bids/bids-mriqc--0.15.0.sing ls -l $d | head -n 1; done
/etc
total 2868
/etc/apt
total 72
/data/famface/
total 16
/home/yoh
total 1317320
/home/yoh/d
FATAL:   container creation failed: mount ->/home/yoh/d error: can't remount /home/yoh/d: no such file or directory
/tmp
total 172088
/tmp/d
FATAL:   container creation failed: mount ->/tmp/d error: can't remount /tmp/d: no such file or directory

in above - /data/famface is not in the container, everything works out just fine. ~/d and /tmp/d -- fail to bind mount

Interestingly, if I provide -W path where I will have those /tmp/d and /home/yoh/d precreated -- seems to "work" for /tmp but not for home:

/tmp/d > mkdir -p {~,/tmp}/d; touch {~,/tmp}/d/file; for d in ~/d /tmp/d; do echo $d ; builtin cd $d; singularity exec -c -W /home/yoh/.tmp/singtmp.HUk5El -B $d /tmp/out1/containers/images/bids/bids-mriqc--0.15.0.sing ls -l $d | head -n 1; done                                               
/home/yoh/d
FATAL:   container creation failed: mount ->/home/yoh/d error: can't remount /home/yoh/d: no such file or directory
/tmp/d
total 0

but you can mention that actually /tmp/d lists no files (total 0) so it doesn't bind mount /tmp/d from outside which has a file in it

PS removed E: could not determine git repository root in the above dumps which were coming from my prompt setup and shouldn't be relevant here

[kyleam]

FWIW I think this matches the issue described by @gmkurtzer on the mailing list:

But it appears we have an issue with that in the version that I'm testing:

$ singularity shell -c -B $PWD alpine.sif FATAL: container creation failed: mount ->/home/gmk/demo error: can't remount /home/gmk/demo: no such file or directory

... Hrmm, that might need a bug report submitted, or someone smarter than me can explain what I'm doing wrong. ;-)

[yarikoptic]

Ha ha, our roads crossed again @gmkurtzer ;-) Thank you @kyleam for digging!

[rherban]

Sharing my findings, though I don't have any concrete explanations.

With -c, a home directory is still being mounted, albeit from a temp location. Trying to mount ~/d in your example is probably throwing mount for a loop since it's already mounted and the underlying directory doesn't exist.

If I do a shell -c -B /home/rherban/tmp test.sif it fails with the same error as you, but if I shell -c -B /home/rherban/tmp:/home/rherban test.sif it works as expected.

@cclerget @ikaneshiro This feels like a bug, but looking for your guidance here.

[jmstover]

When you run with -c ... it creates an empty /tmp and $HOME ... these are overlay tmpfs mount points.

When you try binding a directory from the host to the newly created $HOME in the container with -c, it fails ... because the mount point doesn't exist.

It has to do with the order of operations I believe. It cannot create a subdirectory for the mount if it doesn't exist because the mount point exists on the layer below where the overlay is mounted. So the bind mount point doesn't exist. In other words, the mount operations when in contain mode are I believe:

• Create the bind mount points ** Create /tmp/d and $HOME/d
• Mount the empty Overlay $HOME, /tmp ** We have now covered up /tmp/d and $HOME/d with the empty overlay
• Mount the bind mounts from host ** Bind mount errors, because our mount point doesn't exist

Mounting of the bind mounts happens after the mounting of the empty overlay. So you don't have the mount points existing.

@cclerget ... is there a good way around this to allow running contained, but then still do the bind mounts and create directories within those created overlays? From my understanding there wasn't a good way to do it, is that correct?

[yarikoptic]

Is it really ImageBuilding (as labeled now) or should be "Runtime" label?

[cclerget]

@yarikoptic When you get a chance, could you try #3970

@rherban It's a bug. @jmstover #3970 should (hopefully) fix that