Closed radoye closed 4 years ago
Hi. We need a bit more information here:
singularity inspect lang.sif
from a working machine).uname -a
dmesg
Thanks.
Sure thing, @dctrud.
I'm providing the Lang.def
at the end of this message. I can build containers fine. I have rebuilt this container just before trying to run it. I cannot perform inspect:
$ singularity inspect ~/lang.sif
WARNING: No metadata partition, searching in container...
FATAL: Unable to process command: while running /usr/local/libexec/singularity/bin/starter-suid: exit status 255: FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted
$ uname -a
Linux hopper 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ dmesg
[75591.518372] squashfs: Unknown parameter 'errors'
Container definition
# singularity docks
Bootstrap: docker
#From: tensorflow/tensorflow:2.1.0-gpu-py3-jupyter
#From: nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
From: ubuntu:18.04
%labels
LAYER base
AUTHOR rrs
VERSION v0.3.0
%help
%environment
export LANG='en_US.UTF-8'
export LANGUAGE='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'
export GO_VER=1.13.4
export OS=linux
export ARCH=amd64
export CUDA_VER=10.1
export DEV_PATH=/opt/dev
export RUSTUP_HOME=$DEV_PATH/rust
export CARGO_HOME=$DEV_PATH/rust
export CARGO_BIN=$CARGO_HOME/bin
. $RUSTUP_HOME/env
export HASKELL_HOME=$DEV_PATH/haskell
export HASKELL_BIN=$HASKELL_HOME/bin
export CABAL_HOME=$HASKELL_HOME/cabal
export CABAL_CONFIG=$CABAL_HOME/config
export AGDA_HOME=$HASKELL_HOME/agda
export IDRIS_HOME=$HASKELL_HOME/idris
export ELM_HOME=$HOME
export PATH=$CARGO_BIN:$HASKELL_BIN:$ELM_HOME:/opt/conda/bin:/usr/local/cuda/bin:/usr/local/go/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64
export OCAML_HOME=$DEV_PATH/ocaml
%post
# build the setup
#
# https://sylabs.io/guides/3.5/user-guide/persistent_overlays.html
#
################################################################################
# BASIC SYSTEM
########################################
# infrastructure
apt update -y && \
apt install -y --no-install-recommends software-properties-common && \
add-apt-repository ppa:kelleyk/emacs && \
apt update -y && apt upgrade -y && \
apt install -y --no-install-recommends \
build-essential \
default-jre \
dialog \
cmake \
make \
gcc \
g++ \
curl \
wget \
gnupg \
sudo \
ca-certificates \
libjpeg-dev \
libpng-dev \
libev-dev \
libxkbcommon-x11-0 \
redis-server \
emacs26 \
libopenmpi-dev \
pkg-config \
openssh-client \
zip unzip \
git \
feh \
locales \
bubblewrap
########################################
# basic OS dev tools
TZ=America/Los_Angeles
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && \
apt update -y --fix-missing && \
DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends tzdata && \
apt install -y --no-install-recommends \
python ruby ruby-dev gem xclip \
libnss3-dev libgtk-3-0 libxtst6 libxss-dev libxss1 \
libgraphviz-dev graphviz
#################################################################################
# sublime text
wget -qO - https://download.sublimetext.com/sublimehq-pub.gpg | sudo apt-key add -
sudo apt-add-repository "deb https://download.sublimetext.com/ apt/stable/"
apt install -y --no-install-recommends \
sublime-text
#################################################################################
locale-gen en_US.UTF-8
################################################################################
# LANGUAGES and DEV TOOLS
export DEV_PATH=/opt/dev
########################################
# Node + Optic API stuff
curl -sL https://deb.nodesource.com/setup_12.x | bash && sudo apt update -y
apt install -y nodejs libsecret-1-dev libssl-dev
npm config set python /usr/bin/python
npm install -g gulp gulp-cli
npm install @useoptic/cli -g
apt autoremove -y && apt clean
#################################################################################
# OCaml
add-apt-repository ppa:avsm/ppa && apt update -y && \
apt install -y --no-install-recommends opam m4
# NOTES and links
# https://github.com/ocaml/infrastructure/wiki/Containers#package-sandboxing
# users will have to redo all this
# init with --disable-sandboxing
# https://github.com/janestreet/install-ocaml
opam init
opam update -uy
opam install -y core merlin utop ocp-indent
opam install -y async core js_of_ocaml js_of_ocaml-ppx merlin utop ocp-indent
####################################################
# ELM
export ELM_HOME=$DEV_PATH/elm
mkdir -p $ELM_HOME
cd $ELM_HOME
curl -L -o elm.gz https://github.com/elm/compiler/releases/download/0.19.1/binary-for-linux-64-bit.gz
gunzip elm.gz
chmod +x elm
npm install -g elm-test elm-oracle elm-format @elm-tooling/elm-language-server
###################################################
# Haskell
# https://cabal.readthedocs.io/en/latest/index.html
export HASKELL_HOME=$DEV_PATH/haskell
export HASKELL_BIN=$HASKELL_HOME/bin
export CABAL_HOME=$HASKELL_PATH/cabal
export CABAL_CONFIG=$CABAL_PATH/config
mkdir -p $HASKELL_HOME
mkdir -p $HASKELL_BIN
mkdir -p $CABAL_HOME
apt install cabal-install -y --no-install-recommends
cabal user-config update
cabal update
apt install -y --no-install-recommends \
libicu-dev libtinfo-dev libgmp-dev netbase
curl -sSL https://get.haskellstack.org/ | sh
###################################################
# Idris
export IDRIS_HOME=$HASKELL_HOME/idris
mkdir -p $IDRIS_HOME
cd $IDRIS_HOME
cabal sandbox init
cabal install idris --prefix=$HASKELL_HOME --bindir=$HASKELL_BIN
###################################################
# Agda
# export AGDA_HOME=$HASKELL_HOME/agda
#
# mkdir -p $AGDA_HOME
# cd $AGDA_HOME
# cabal sandbox init
# cabal install 'alex >=3.1.0 && <3.2.0 || ==3.2.1 || >=3.2.3' --prefix=$HASKELL_HOME --bindir=$HASKELL_BIN
# cabal install alex happy Agda --prefix=$HASKELL_HOME --bindir=$HASKELL_BIN
###################################################
# Rust
# https://doc.rust-lang.org/book/ch01-01-installation.html#installing-rustup-on-linux-or-macos
export RUSTUP_HOME=$DEV_PATH/rust
export CARGO_HOME=$DEV_PATH/rust
export CARGO_BIN=$CARGO_HOME/bin
mkdir -p $RUSTUP_HOME
mkdir -p $CARGO_HOME
mkdir -p $CARGO_BIN
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
. $CARGO_HOME/env
#################################################################################
# make everything writable
chmod -R a+rwx $DEV_PATH
%files
# copy local files
%startscript
# in the container; instance creation
%runscript
# in the container; run
%test
# run test code
@radoye - this is same as #4801
[75591.518372] squashfs: Unknown parameter 'errors'
It was fixed in PR #4802 which is available in Singularity 3.5.1 and newer. Please try a newer version and let us know if you still have an issue.
I'll close this issue as the fix has already been released.
Not sure if this is the same issue but this is localized to one compute node. There's something off with/proc/self/fd//3
:
ls -l /proc/self/fd/
total 0
lrwx------ 1 root root 64 Mar 26 12:37 0 -> /dev/pts/14
lrwx------ 1 root root 64 Mar 26 12:37 1 -> /dev/pts/14
lrwx------ 1 root root 64 Mar 26 12:37 2 -> /dev/pts/14
lr-x------ 1 root root 64 Mar 26 12:37 3 -> /proc/3279363/fd
They don't exist:
# ls -l /proc/self/fd/3
ls: cannot access '/proc/self/fd/3': No such file or directory
# ls -l /proc/3279363/fd
ls: cannot access '/proc/3279363/fd': No such file or directory
A reboot will likely fix this but is there another option?
apptainer --version
apptainer version 1.3.0-1.el9
apptainer -d shell /local/jupyterlab_torch22_lightning22_fa2-2024-03-05-c636d2f73ed3.sif
DEBUG [U=0,P=3270343] persistentPreRun() Apptainer version: 1.3.0-1.el9
DEBUG [U=0,P=3270343] persistentPreRun() Parsing configuration file /etc/apptainer/apptainer.conf
DEBUG [U=0,P=3270343] SetBinaryPath() Setting binary path to /usr/libexec/apptainer/bin:/cm/shared/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/root/.local/bin:/root/bin:/cm/local/apps/environment-modules/4.5.3//bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/cm/local/apps/environment-modules/4.5.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=0,P=3270343] SetBinaryPath() Using that path for all binaries
DEBUG [U=0,P=3270343] handleConfDir() /root/.apptainer already exists. Not creating.
DEBUG [U=0,P=3270343] setUmask() Saving umask 0022 for propagation into container
DEBUG [U=0,P=3270343] checkEncryptionKey() Checking for encrypted system partition
DEBUG [U=0,P=3270343] Init() Image format detection
DEBUG [U=0,P=3270343] Init() Check for sandbox image format
DEBUG [U=0,P=3270343] Init() sandbox format initializer returned: not a directory image
DEBUG [U=0,P=3270343] Init() Check for sif image format
DEBUG [U=0,P=3270343] Init() sif image format detected
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MODULES_SET_SHELL_STARTUP environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SUDO_GID environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding HISTCONTROL environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding HOSTNAME environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding HISTSIZE environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding _LMFILES__modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LIBRARY_PATH_modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SUDO_COMMAND environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding CPATH_modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SUDO_USER environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MANPATH_modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MODULESHOME environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LD_LIBRARY_PATH_modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding PATH_modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LOADEDMODULES_modshare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LESSOPEN environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LIBRARY_PATH environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding LOADEDMODULES environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding S_COLORS environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding PS1 environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding CMD_WLM_CLUSTER_NAME environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding ENABLE_LMOD environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding which_declare environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SLURM_CONF environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MODULEPATH environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding _LMFILES_ environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding SUDO_UID environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding CPATH environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding MODULES_CMD environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding BASH_FUNC_ml%% environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding BASH_FUNC_switchml%% environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding BASH_FUNC_which%% environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding BASH_FUNC_module%% environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding BASH_FUNC__module_raw%% environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding _ environment variable
VERBOSE [U=0,P=3270343] SetContainerEnv() Not forwarding APPTAINER_DEBUG environment variable
DEBUG [U=0,P=3270343] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=0,P=3270343] SetContainerEnv() Setting HOME=/root
VERBOSE [U=0,P=3270343] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=0,P=3270343] InitImageDrivers() Skipping installing fuseapps image driver because running as root
DEBUG [U=0,P=3270343] SetuidMountAllowed() Kernel squashfs mount allowed because running as root
DEBUG [U=0,P=3270343] init() Use starter binary /usr/libexec/apptainer/bin/starter
VERBOSE [U=0,P=3270343] print() Set messagelevel to: 5
VERBOSE [U=0,P=3270343] init() Starter initialization
VERBOSE [U=0,P=3270343] is_suid() Check if we are running as setuid: 0
DEBUG [U=0,P=3270343] read_engine_config() Read engine configuration
DEBUG [U=0,P=3270343] init() Wait completion of stage1
DEBUG [U=0,P=3270357] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=3270357] init() Spawn stage 1
DEBUG [U=0,P=3270357] func1() executablePath is /usr/libexec/apptainer/bin/starter
DEBUG [U=0,P=3270357] func1() starter was not relocated from /usr/libexec
DEBUG [U=0,P=3270357] func1() Install prefix is /usr
DEBUG [U=0,P=3270357] startup() apptainer runtime engine selected
VERBOSE [U=0,P=3270357] startup() Execute stage 1
DEBUG [U=0,P=3270357] StageOne() Entering stage 1
DEBUG [U=0,P=3270357] InitImageDrivers() Skipping installing fuseapps image driver because running as root
DEBUG [U=0,P=3270357] prepareRootCaps() Root full capabilities
DEBUG [U=0,P=3270357] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=0,P=3270357] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=0,P=3270357] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=0,P=3270357] prepareAutofs() Could not keep file descriptor for home directory /root: no mount point
DEBUG [U=0,P=3270357] prepareAutofs() Could not keep file descriptor for current working directory /root: no mount point
DEBUG [U=0,P=3270357] Init() Image format detection
DEBUG [U=0,P=3270357] Init() Check for sandbox image format
DEBUG [U=0,P=3270357] Init() sandbox format initializer returned: not a directory image
DEBUG [U=0,P=3270357] Init() Check for sif image format
DEBUG [U=0,P=3270357] Init() sif image format detected
DEBUG [U=0,P=3270357] setSessionLayer() Using overlay because it is not disabled
DEBUG [U=0,P=3270357] PrepareConfig() image driver is
VERBOSE [U=0,P=3270343] wait_child() stage 1 exited with status 0
DEBUG [U=0,P=3270343] cleanup_fd() Close file descriptor 4
DEBUG [U=0,P=3270343] cleanup_fd() Close file descriptor 5
DEBUG [U=0,P=3270343] cleanup_fd() Close file descriptor 6
DEBUG [U=0,P=3270343] init() Set child signal mask
DEBUG [U=0,P=3270343] init() Create socketpair for master communication channel
DEBUG [U=0,P=3270343] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=0,P=3270343] init() Spawn master process
DEBUG [U=0,P=3270365] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=3270365] create_namespace() Create mount namespace
VERBOSE [U=0,P=3270343] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=3270343] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=0,P=3270365] create_namespace() Create mount namespace
VERBOSE [U=0,P=3270366] init() Spawn RPC server
DEBUG [U=0,P=3270343] func1() executablePath is /usr/libexec/apptainer/bin/starter
DEBUG [U=0,P=3270343] func1() starter was not relocated from /usr/libexec
DEBUG [U=0,P=3270343] func1() Install prefix is /usr
DEBUG [U=0,P=3270366] func1() executablePath is /usr/libexec/apptainer/bin/starter
DEBUG [U=0,P=3270366] func1() starter was not relocated from /usr/libexec
DEBUG [U=0,P=3270366] func1() Install prefix is /usr
DEBUG [U=0,P=3270343] startup() apptainer runtime engine selected
VERBOSE [U=0,P=3270343] startup() Execute master process
DEBUG [U=0,P=3270366] startup() apptainer runtime engine selected
VERBOSE [U=0,P=3270366] startup() Serve RPC requests
DEBUG [U=0,P=3270343] InitImageDrivers() Skipping installing fuseapps image driver because running as root
DEBUG [U=0,P=3270343] setupSessionLayout() Using Layer system: overlay
DEBUG [U=0,P=3270343] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=0,P=3270343] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=0,P=3270343] addRootfsMount() Image type is 4096
DEBUG [U=0,P=3270343] addRootfsMount() Mounting block [squashfs] image: /local/jupyterlab_torch22_lightning22_fa2-2024-03-05-c636d2f73ed3.sif
DEBUG [U=0,P=3270343] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=0,P=3270343] addKernelMount() Adding proc to mount list
VERBOSE [U=0,P=3270343] addKernelMount() Default mount: /proc:/proc
DEBUG [U=0,P=3270343] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=0,P=3270343] addKernelMount() Adding sysfs to mount list
VERBOSE [U=0,P=3270343] addKernelMount() Default mount: /sys:/sys
DEBUG [U=0,P=3270343] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=0,P=3270343] addDevMount() Adding dev to mount list
VERBOSE [U=0,P=3270343] addDevMount() Default mount: /dev:/dev
DEBUG [U=0,P=3270343] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=0,P=3270343] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=0,P=3270343] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=0,P=3270343] addHomeStagingDir() Staging home directory (/root) at /var/lib/apptainer/mnt/session/root
DEBUG [U=0,P=3270343] addHomeMount() Adding home directory mount [/var/lib/apptainer/mnt/session/root:/root] to list using layer: overlay
DEBUG [U=0,P=3270343] addTmpMount() Checking for 'mount tmp' in configuration file
DEBUG [U=0,P=3270343] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=0,P=3270343] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=0,P=3270343] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=0,P=3270343] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=0,P=3270343] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=0,P=3270343] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=0,P=3270343] create() Mount all
DEBUG [U=0,P=3270343] mountGeneric() Mounting tmpfs to /var/lib/apptainer/mnt/session
DEBUG [U=0,P=3270343] mountImage() Mounting loop device /dev/loop0 to /var/lib/apptainer/mnt/session/rootfs of type squashfs
DEBUG [U=0,P=3270343] CleanupContainer() Cleanup container
FATAL [U=0,P=3270343] Master() container creation failed: mount hook function failure: mount /proc/self/fd/3->/var/lib/apptainer/mnt/session/rootfs error: while mounting image /proc/self/fd/3: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted
This repository is closed. Please create a new issue in https://github.com/apptainer/apptainer with all your OS particulars
Version of Singularity:
What version of Singularity are you using? Run:
A container that previously used to work (pre Ubuntu update) suddenly does not work. Rebuilding the container does not help.
Expected behavior
Container starts.
Actual behavior
Error is printed out.
Steps to reproduce this behavior
Updated Ubuntu 18.04 yesterday.
What OS/distro are you running
How did you install Singularity
Source.