search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

image compressed with xz :: failed to mount squashfs filesystem #3439

Closed EricDeveaud closed 5 years ago

EricDeveaud commented 5 years ago

Version of Singularity:

singularity 3.1.0 buit with squashfs with default compression xz instead of gzip

Expected behavior

build and run image as usual

Actual behavior

see bellow

Steps to reproduce behavior

1) build squashfs with COMP_DEFAULT = xz 2) build singularity/3.1.0 with this one and set singularity.conf to point to mksquasfs with compression default = xz NB no system mksquashfs available on the computer other than this one 

bigmess:/tmp > grep "mksquashfs path" /soft/singularity/3.1.0.xz/etc/singularity/singularity.conf
mksquashfs path = /soft/squashfs/4.3.xz/bin/mksquashfs

and we have 

bigmess:/tmp > /soft/squashfs/4.3.xz/bin/mksquashfs |& head     
SYNTAX:/soft/squashfs/4.3.xz/bin/mksquashfs source1 source2 ...  dest [options] [-e list of exclude
dirs/files]

Filesystem build options:
-comp <comp>            select <comp> compression
                        Compressors available:
                                gzip
                                lzo
                                xz (default)
-b <block_size>         set data block to <block_size>.  Default 128 Kbytes

build of the image is possible 

bigmess:/tmp > load singularity/3.1.0.xz
bigmess:/tmp > sudo singularity build img.xz.simg recipe
[sudo] password for edeveaud: 
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df
Copying config sha256:b9a1b1f0b2baaec83946a26d7045e4028f11eccc8b0e5b3514568e56a391beb2
 1.05 KiB / 1.05 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: img.xz.simg

but it's not possible to run it ;-(

bigmess:/tmp > singularity run img.xz.simg
FATAL:   container creation failed: mount error: can't mount image /proc/self/fd/5: failed to mount squashfs filesystem: invalid argument

nor inspect it, not help 

bigmess:/tmp > singularity inspect img.xz.simg                            
FATAL:   exit status 255

when using singularity built versus mksquashfs with COMP_DEFAULT = gzip everything run smoothly:

bigmess:/tmp > module load singularity/3.1.0
bigmess:/tmp > sudo singularity  build img.simg recipe         
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Skipping fetch of repeat blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df
Copying config sha256:b9a1b1f0b2baaec83946a26d7045e4028f11eccc8b0e5b3514568e56a391beb2
 1.05 KiB / 1.05 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: img.simg

and

bigmess:/tmp > singularity run img.simg    
Hello!

regards

Eric

jscook2345 commented 5 years ago

@EricDeveaud

Can you run your examples against the xz compressed image with the -d option?

ex: singulairty -d inspect img.xz.simg

Also, what OS version are you using?

EricDeveaud commented 5 years ago

@jscook2345 here are the info requested.

OS: CentOS release 6.10 (Final)

here is the recipe used 

bigmess:/tmp > cat recipe 
BootStrap: docker
From: centos
%runscript
    echo "Hello!"

building with singularity/3.1.0 with mksquashfs with COMP_DEFAULT=xz

bigmess:/tmp > sudo singularity -d build  img.xz.simg recipe
[sudo] password for edeveaud: 
WARNING [U=0,P=15177]      sylabsToken()                 Authentication token file not found : Only pulls of public images will succeed
DEBUG   [U=0,P=15177]      Init()                        Entering image format intializer
DEBUG   [U=0,P=15177]      Init()                        Check for image format sif
DEBUG   [U=0,P=15177]      Init()                        sif format initializer returns: can't read first 2048 bytes: %!s(<nil>)
DEBUG   [U=0,P=15177]      Init()                        Check for image format sandbox
DEBUG   [U=0,P=15177]      Init()                        sandbox format initializer returns: not a directory image
DEBUG   [U=0,P=15177]      Init()                        Check for image format squashfs
DEBUG   [U=0,P=15177]      Init()                        squashfs format initializer returns: can't read first 2048 bytes: %!s(<nil>)
DEBUG   [U=0,P=15177]      Init()                        Check for image format ext3
DEBUG   [U=0,P=15177]      Init()                        ext3 format initializer returns: can't read first 2048 bytes: %!s(<nil>)
DEBUG   [U=0,P=15177]      NewBundle()                   Created temporary directory for bundle /tmp/sbuild-585114607
INFO    [U=0,P=15177]      Full()                        Starting build...
DEBUG   [U=0,P=15177]      Get()                         Reference: centos
DEBUG   [U=0,P=15177]      updateCacheSubdir()           Caching directory set to /root/.singularity/cache/oci
Getting image source signatures
Skipping fetch of repeat blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df
Copying config sha256:b9a1b1f0b2baaec83946a26d7045e4028f11eccc8b0e5b3514568e56a391beb2
 1.05 KiB / 1.05 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
DEBUG   [U=0,P=15177]      Full()                        Inserting Metadata
INFO    [U=0,P=15177]      insertRunScript()             Adding runscript
DEBUG   [U=0,P=15177]      Full()                        Calling assembler
INFO    [U=0,P=15177]      Assemble()                    Creating SIF file...
INFO    [U=0,P=15177]      Full()                        Build complete: img.xz.simg
DEBUG   [U=0,P=15177]      cleanUp()                     Build bundle cleanup: /tmp/sbuild-585114607

now running 

bigmess:/tmp > singularity -d run  img.xz.simg 
VERBOSE [U=0,P=15293]      print()                       Set messagelevel to: 5
DEBUG   [U=0,P=15293]      init()                        PIPE_EXEC_FD value: 9
VERBOSE [U=0,P=15293]      init()                        Container runtime
VERBOSE [U=0,P=15293]      is_suid()                     Check if we are running as setuid
DEBUG   [U=0,P=15293]      init()                        Overlay seems not supported by kernel
DEBUG   [U=0,P=15293]      init()                        Drop privileges
DEBUG   [U=7225,P=15293]   init()                        Read json configuration from pipe
DEBUG   [U=7225,P=15293]   init()                        Set child signal mask
DEBUG   [U=7225,P=15293]   init()                        Wait completion of stage1
VERBOSE [U=7225,P=15305]   priv_escalate()               Get root privileges
DEBUG   [U=0,P=15305]      set_parent_death_signal()     Set parent death signal to 9
DEBUG   [U=0,P=15305]      prepare_stage()               Entering in stage 1
DEBUG   [U=7225,P=15305]   set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=7225,P=15305]   init()                        Spawn stage 1
VERBOSE [U=7225,P=15305]   startup()                     Execute stage 1
DEBUG   [U=7225,P=15305]   Stage()                       Entering stage 1
DEBUG   [U=7225,P=15305]   Init()                        Entering image format intializer
DEBUG   [U=7225,P=15305]   Init()                        Check for image format sif
DEBUG   [U=7225,P=15293]   init()                        Create socketpair for master communication channel
DEBUG   [U=7225,P=15293]   cleanup_fd()                  Check file descriptor /proc/self/fd/3 pointing to /dev/urandom
DEBUG   [U=7225,P=15293]   cleanup_fd()                  Check file descriptor /proc/self/fd/4 pointing to [eventpoll]
DEBUG   [U=7225,P=15293]   cleanup_fd()                  Check file descriptor /proc/self/fd/5 pointing to /tmp/img.xz.simg
DEBUG   [U=7225,P=15293]   cleanup_fd()                  Check file descriptor /proc/self/fd/6 pointing to /var/lib/sss/mc/passwd
DEBUG   [U=7225,P=15293]   cleanup_fd()                  Check file descriptor /proc/self/fd/7 pointing to /var/lib/sss/mc/group
VERBOSE [U=7225,P=15293]   priv_escalate()               Get root privileges
VERBOSE [U=0,P=15293]      create_namespace()            Create mount namespace
DEBUG   [U=0,P=15293]      init()                        Create RPC socketpair for communication between stage 2 and RPC server
DEBUG   [U=0,P=15293]      set_terminal_control()        Pass terminal control to child
VERBOSE [U=0,P=15293]      init()                        Spawn master process
DEBUG   [U=0,P=15312]      set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=15312]      create_namespace()            Create mount namespace
DEBUG   [U=0,P=15312]      set_parent_death_signal()     Set parent death signal to 9
DEBUG   [U=0,P=15312]      prepare_stage()               Entering in stage 2
VERBOSE [U=0,P=15313]      init()                        Spawn RPC server
DEBUG   [U=7225,P=15312]   set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=7225,P=15293]   startup()                     Execute master process
VERBOSE [U=0,P=15313]      startup()                     Serve RPC requests
DEBUG   [U=7225,P=15293]   setupSessionLayout()          Attempting to use underlay (enable underlay = yes)
DEBUG   [U=7225,P=15293]   setupUnderlayLayout()         Creating underlay SESSIONDIR layout
DEBUG   [U=7225,P=15293]   addRootfsMount()              Mount rootfs in read-only mode
DEBUG   [U=7225,P=15293]   addRootfsMount()              Mounting block [squashfs] image: /tmp/img.xz.simg
DEBUG   [U=7225,P=15293]   addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=7225,P=15293]   addKernelMount()              Adding proc to mount list
VERBOSE [U=7225,P=15293]   addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=7225,P=15293]   addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=7225,P=15293]   addKernelMount()              Adding sysfs to mount list
VERBOSE [U=7225,P=15293]   addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=7225,P=15293]   addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=7225,P=15293]   addDevMount()                 Adding dev to mount list
VERBOSE [U=7225,P=15293]   addDevMount()                 Default mount: /dev:/dev
DEBUG   [U=7225,P=15293]   addHostMount()                Not mounting host file systems per configuration
VERBOSE [U=7225,P=15293]   addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=7225,P=15293]   addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=7225,P=15293]   addHomeMount()                Skipping home dir mounting (per config)
DEBUG   [U=7225,P=15293]   addTmpMount()                 Checking for 'mount tmp' in configuration file
VERBOSE [U=7225,P=15293]   addTmpMount()                 Default mount: /tmp:/tmp
VERBOSE [U=7225,P=15293]   addTmpMount()                 Default mount: /var/tmp:/var/tmp
DEBUG   [U=7225,P=15293]   addScratchMount()             Not mounting scratch directory: Not requested
VERBOSE [U=7225,P=15293]   addCwdMount()                 Default mount: /tmp: to the container
DEBUG   [U=7225,P=15293]   addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=7225,P=15293]   addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=7225,P=15293]   addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=7225,P=15293]   addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=7225,P=15293]   create()                      Mount all
DEBUG   [U=7225,P=15293]   mountGeneric()                Mounting tmpfs to /mount/gensoft2/exe/singularity/3.1.0.xz/var/singularity/mnt/session
DEBUG   [U=7225,P=15293]   mountImage()                  Mounting loop device /dev/loop2 to /mount/gensoft2/exe/singularity/3.1.0.xz/var/singularity/mnt/session/rootfs
DEBUG   [U=7225,P=15293]   CleanupContainer()            Cleanup container
FATAL   [U=7225,P=15293]   Master()                      container creation failed: mount error: can't mount image /proc/self/fd/5: failed to mount squashfs filesystem: invalid argument
bigmess:/tmp >

inspect

bigmess:/tmp > singularity -d inspect  img.xz.simg 
DEBUG   [U=7225,P=15339]   func13()                      Inspection of labels as default.
FATAL   [U=7225,P=15339]   getFileContent()              exit status 255

hope this will gives you better informations.

regards

Eric

jmstover commented 5 years ago

OS: CentOS release 6.10 (Final)

RHEL 6 only supports gzip compression. The Kernel doesn't have support for anything else.

Edit: I've created an image on my Slackware laptop that defaults to XZ compression, and that image would not run on a CentOS 6.10 VM since the Kernel doesn't have support for it. CentOS 7 it does work on.

EricDeveaud commented 5 years ago

@jmstover thanks for the higlight.

I was thinking that unsquash was performed thru squasfh tools (eg unsquashfs). so is the kernel that play the unsquash role. where I can find doc about this

regards

Eric

jmstover commented 5 years ago

@EricDeveaud ,

When you run an Image, you aren't unsquashfs'ing it, but it gets mounted on a loop device and then read. This is where the Kernel squashfs module comes into play.

In the History section here you can see some of the compression history: https://en.wikipedia.org/wiki/SquashFS

I think that the Kernel itself needs to support the compression algorithm, so you could have a new enough Kernel, but not have support for that compression type available to you if it was left out when the Kernel was built.

-J

truatpasteurdotfr commented 5 years ago

on c6

[tru@tars-submit0 ~]$ egrep -e 'GZIP|XZ' /boot/config-2.6.32-754.11.1.el6.x86_64 
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_KERNEL_GZIP=y
CONFIG_RD_GZIP=y
CONFIG_DECOMPRESS_GZIP=y

on c7

[tru@sillage ~]$  egrep -e 'GZIP|XZ' /boot/config-3.10.0-957.12.2.el7.x86_64 
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_XZ is not set
CONFIG_RD_GZIP=y
CONFIG_RD_XZ=y
CONFIG_SQUASHFS_XZ=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
# CONFIG_XZ_DEC_IA64 is not set
# CONFIG_XZ_DEC_ARM is not set
# CONFIG_XZ_DEC_ARMTHUMB is not set
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_XZ=y
mem commented 5 years ago

Maybe @ikaneshiro can provide some info.

ikaneshiro commented 5 years ago

Since https://github.com/sylabs/singularity/pull/3925 is merged I think we can close this issue because we are only supporting gzip compression on images in order to use the most common algorithm to avoid problems like this. Feel free to reopen this or open a new issue if this needs further discussion.