Singularity container - difficulty building flag_combinefilter.image

[G-Thomson]

I know you have been working on packaging FLAG in one Singularity container. I have been trying to run it today on my institution's cluster and have been running into issues.

First, when I ran build_singularity_flag.sh I got the following error (or similar) for every sub singularity container called in direct_pull_singularity_images_and_move_to_folders.sh

FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/seqkit: apply hdr metadata: restore chown metadata: /tmp/build-temp-3056869678/rootfs/opt/seqkit: lchown /tmp/build-temp-3056869678/rootfs/opt/seqkit: invalid argument

I solved this by amending ./containers/singularity_flag/singularity_flag.def to use a slightly newer version of singularity wget -O- https://github.com/sylabs/singularity/releases/download/v4.1.2/singularity-ce_4.1.2-jammy_amd64.deb > singularity.deb

and by directing singularity tmp files to the /tmp directory for each pull command in direct_pull_singularity_images_and_move_to_folders.sh via the following command:

sed -i -e 's/pull/pull --tmpdir=\/tmp/g' direct_pull_singularity_images_and_move_to_folders.sh

I an now running into an issue in building the flag_combinefilter.image file. I get the following error.

WARNING: Could not lookup the current user's information: user: unknown userid 16748
FATAL:   container creation failed: mount /.singularity.d/libs/fakeroot->/.singularity.d/libs/fakeroot error: while mounting /.singularity.d/libs/fakeroot: destination /.singularity.d/libs/fakeroot doesn't exist in container
FATAL:   While performing build: while running engine: exit status 255

I am not yet sure what this means or how to fix it.

Many thanks for your help.

[wtroy2]

Hey, ya I'm currently working on it and testing as a non root user. I should have a branch of it up tomorrow for people to test. I'll let you know once it's up!

On Fri, Apr 19, 2024, 5:36 PM Geoffrey Thomson @.***> wrote:

I know you have been working on packaging FLAG in one Singularity container. I have been trying to run it today on my institution's cluster and have been running into issues.

First, when I ran build_singularity_flag.sh I got the following error (or similar) for every sub singularity container called in direct_pull_singularity_images_and_move_to_folders.sh

FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: opt/seqkit: apply hdr metadata: restore chown metadata: /tmp/build-temp-3056869678/rootfs/opt/seqkit: lchown /tmp/build-temp-3056869678/rootfs/opt/seqkit: invalid argument

I solved this by amending ./containers/singularity_flag/singularity_flag.def to use a slightly newer version of singularity wget -O- https://github.com/sylabs/singularity/releases/download/v4.1.2/singularity-ce_4.1.2-jammy_amd64.deb

singularity.deb

and by directing singularity tmp files to the /tmp directory for each pull command in direct_pull_singularity_images_and_move_to_folders.sh via the following command:

sed -i -e 's/pull/pull --tmpdir=\/tmp/g' direct_pull_singularity_images_and_move_to_folders.sh

I an now running into an issue in building the flag_combinefilter.image file. I get the following error.

WARNING: Could not lookup the current user's information: user: unknown userid 16748 FATAL: container creation failed: mount /.singularity.d/libs/fakeroot->/.singularity.d/libs/fakeroot error: while mounting /.singularity.d/libs/fakeroot: destination /.singularity.d/libs/fakeroot doesn't exist in container FATAL: While performing build: while running engine: exit status 255

I am not yet sure what this means or how to fix it.

Many thanks for your help.

— Reply to this email directly, view it on GitHub https://github.com/formbio/FLAG/issues/20, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHT22X2NHQ2JMJPHKGMWFCDY6GL7BAVCNFSM6AAAAABGP5MK4SVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TIMJUGAYDCMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[wtroy2]

@G-Thomson the update is up on the main branch. Running bash build_singularity_flag.sh in the home directory of the repo should setup everything you need to run it in the examples directory. From there download the genome file https://ftp.ensembl.org/pub/rapid-release/species/Erynnis_tages/GCA_905147235.1/braker/genome/Erynnis_tages-GCA_905147235.1-softmasked.fa.gz in the examples directory and ungzip the example files and fingers crossed it will work for you.

Please keep me updated if you have any issues. Getting it to run for people with singularity has been more complicated than Ive been expecting it to be.

[G-Thomson]

Hi @wtroy2. Thank you for the effort you are putting in to get this working.

I ran the build_singularity_flag.sh and I got the following:

+ cd /opt/FLAG/containers/augustus/
+ singularity pull flag_augustus.image docker://ghcr.io/formbio/flag_augustus_singularity:latest
^[[33mWARNING:^[[0m Could not lookup the current user's information: user: unknown userid 16748
INFO:    Converting OCI blobs to SIF format
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack entry: var/cache/apt/arch$
FATAL:   While performing build: while running engine: exit status 255
Entering the examples directory
Moving the singularity_flag singularity image to the examples directory
mv: cannot stat '../containers/singularity_flag/singularity_flag.image': No such file or directory

It did not go further then that and try and pull the other singularity containers.

Also, before I realized that the setup had failed I tried running it and I got :

INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    Using fakeroot command combined with root-mapped namespace
FATAL:   While checking container encryption: could not open image /home/got3/project/CarrizoGenome/flag/FLAG/examples/singularity_flag_new.image: failed to retrieve path for /home/got3/project/CarrizoGenome/flag/FLAG/examples/singularity_flag_new.image: lstat /gpfs/gibbs/project/jacob/got3/CarrizoGenome/flag/FLAG/examples/singularity_flag_new.image: no such file or directory

I don't know if the first two lines are related too it not creating the singularity_flag_new.image in the first place but I thought they might be helpful.

The build_singularity_flag.sh also did not create an output directory.

[wtroy2]

it may be because of tmp directory permission you have. Im unsure. Try pulling the changes to the main now and building with the script

[G-Thomson]

Unfortunately that didn't fix it.

I notice that when using docker there are individual containers for each program. What is the issue that requires bundling them all together when using Singularity?

[wtroy2]

It was originally like that but some people had trouble pulling certain images, seemingly depending on their singularity version, and then others had trouble running nextflow so was trying to put it all in one with a single singularity version to get around this but seems like this approach may also be having issues.

What version of singularity are you running? Ill try to test on that and a couple of other versions to see if its related to that

[wtroy2]

I ran my original tests on singularity v4.1. Now testing on 2.5 and ya it doesn't work. Pulling from docker doesn't even work on this version it seems...

[G-Thomson]

I am using apptainer version 1.2.5-1.el8