Error while building from image: "failed to create symlink /image/rootfs/var/tmp"

[Flamefire]

Version of Singularity

singularity-ce version 4.1.4-jammy

Describe the bug

I cannot use a (certain class of) images to build from. In particular I need to encrypt existing containers for use in sensitive environments. I cannot recreate the existing container, i.e. I need to use the image file.

However the build fails with

FATAL:   While performing build: packer failed to pack: root filesystem extraction failed: extract command failed:
[some "WARNING:" lines]

create_inode: failed to create symlink /image/rootfs/var/tmp, because File exists

To Reproduce

Intended command is: singularity build --pem-path=/keys/image.pub output.img input.img, but any build fails:

Steps to reproduce the behavior:

• wget -O input.img https://depot.galaxyproject.org/singularity/pigz%3A2.3.4
• sudo singularity build --sandbox /tmp/img input.img

Expected behavior

Encrypted image (or sandbox folder) is created

OS / Linux Distribution

Which Linux distribution are you using? Is it up-to-date?

Linux Mint 21.3 (Ubuntu 22.04 "jammy") fully patched

Installation Method

DEB from GitHub: singularity-ce_4.1.4-jammy_amd64.deb

Additional context

Log with --debug

Console output

INFO:    Starting build...
FATAL:   While performing build: packer failed to pack: root filesystem extraction failed: extract command failed: WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
WARNING: Skipping mount /var/lib/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
WARNING: Skipping mount /var/lib/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
WARNING: Skipping mount /var/lib/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container

create_inode: failed to create symlink /image/rootfs/var/tmp, because File exists
Parallel unsquashfs: Using 4 processors
391 inodes (635 blocks) to write

[==============================================================| ] 634/635  99%

created 109 files
created 54 directories
created 281 symlinks
created 0 devices
created 0 fifos
created 0 sockets
: exit status 2

[dtrudg]

Thanks for the report. We will have to look into this.

@cyanezstange - please could you try to replicate the issue that has been reported, and take a quick look for anything obvious... i.e. whether the image has complex symlink structures pointing to /tmp etc.

[Flamefire]

Another image from the same site works: https://depot.galaxyproject.org/singularity/p7zip%3A16.02

In my test of 77 images from that site I found 58 working and 19 not working.

Some working ones:

• ensembl-vep:110.0--pl5321h2a3209d_0
• fastp:0.23.4--h5f740d0_0
• fastqc:0.12.1--hdfd78af_0
• fgbio:2.0.2--hdfd78af_0
• freebayes:1.3.6--hbfe0e7f_2
• gatk4:4.4.0.0--py36hdfd78af_0
• gatk4-spark:4.4.0.0--hdfd78af_0
• khmer:3.0.0a3--py37haa7609a_2
• macs2:2.2.7.1--py38h4a8c8d9_3
• manta:1.6.0--h9ee0642_1
• mosdepth:0.3.3--hdfd78af_1
• msisensor-pro:1.2.0--hfc31af2_0
• mulled-v2:0560a8046fc82aa4338588eca29ff18edab2c5aa-5687a7da26983502d0a8a9a6b05ed727c740ddc4-0
• mulled-v2:19fa9f1a5c3966b63a24166365e81da35738c5ab-cee56b506ceb753d4bbef7e05b81e1bfc25d937f-0
• mulled-v2:1f09f39f20b1c4ee36581dc81cc323c70e661633-5b2e433ab8b3d1ef098fc944b567fd98caa23f56-0

Some broken ones:

• mulled-v2-ad9dd5f398966bf899ae05f8e7c54d0fb10cdfa7:05678da05b8e5a7a5130e90a9f9a6c585b965afa-0
• mulled-v2-e5d375990341c5aef3c9aff74f96f66f65375ef6:2cdf6bf1e92acbeb9b2834b1c58754167173a410-0
• mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:219b6c272b25e7e642ae3ff0bf0c5c81a5135ab4-0
• perl:5.26.2
• pigz:2.3.4
• python:3.8.3
• star:2.6.1d--0
• subread:2.0.1--hed695b0_0
• vcftools:0.1.16--he513fc3_4

Especially the "mulled" images might be interesting as some work and some don't.

[dtrudg]

@Flamefire - thanks for the extra information. That will be useful.

[Flamefire]

I have some more: Extracting the image manually I see this:

# ll -a /img/root/var/
insgesamt 16
drwxrwxr-x  4 root     root     4096 Mai 23  2014 ./
drwxr-xr-x 18 root     root     4096 Mär  8  2021 ../
lrwxrwxrwx  1 root     root        6 Feb 27  2014 cache -> ../tmp/
drwxrwxr-x  2 root     root     4096 Feb 27  2014 lib/
lrwxrwxrwx  1 root     root        6 Feb 27  2014 lock -> ../tmp/
lrwxrwxrwx  1 root     root        6 Feb 27  2014 log -> ../tmp/
lrwxrwxrwx  1 root     root        6 Feb 27  2014 pcmcia -> ../tmp/
lrwxrwxrwx  1 root     root        6 Feb 27  2014 run -> ../tmp/
lrwxrwxrwx  1 root     root        6 Feb 27  2014 spool -> ../tmp/
lrwxrwxrwx  1 root     root        6 Feb 27  2014 tmp -> ../tmp/
drwxr-xr-x  2 www-data www-data 4096 Mai 23  2014 www/

So the image links all the /var directories to /tmp, likely such that (only the single) /tmp can be mounted to the host

However the precreated rootfs already creates /var/tmp: https://github.com/sylabs/singularity/blob/914126ad9897ed395b5c95ec3384eb035ef9cc6b/internal/pkg/build/sources/base_environment.go#L297-L299

Removing that line makes it work.

[dtrudg]

@Flamefire - thanks for digging. Looks like the force overwrite in the unsquashfs extraction will not force overwrite the directory with a symlink.

Will have a think about this... we should be able to re-order some things to address it without entirely removing /var/tmp creation on images that don't have it.

[Flamefire]

I had an idea: How about removing all (empty) directories from the target where "something" is present in the image?

See #3105 which includes a test that fails on main and the result is tested against my file.

[dtrudg]

We want to tackle this in the opposite manner... extract the image first, in its entirity, and only add directories if they are missing. This should be simpler, and is already the case when building from some other types of sources.

@cyanezstange is going to be working on this in the next week or so.

[Flamefire]

@cyanezstange I looked into that proposed solution and implemented it. I added a test that verifies it and tested it against one of my files causing trouble: Extraction to a sandbox and repacking (directly) to an encrypted image works with that change.

[Flamefire]

I reported this to squashfs-tools and it was fixed there: https://github.com/plougher/squashfs-tools/commit/31cd4d68c36531ed3a25c5fda583efcf03f4c0de

However only empty directories are removed, so it might still be worth to create the directories after extraction although my test file now works with only the patched unsquashfs.

[dtrudg]

@Flamefire - thanks for reporting upstream. We will still have to work around it in Singularity, as it will take a long time for the fix to reach distributions commonly employed in the HPC world.

[Flamefire]

True, that's the situation here. However squashfs-tools is easier to compile than singularity and you can set the unsquashfs path in the singularity configuration file which provides a nice workaround until the update is there.