search

epi2me-labs / wf-alignment

Other
23 stars 15 forks source link

Issue with containers - .command.sh: line 2: python: command not found #33

Open Erin-Chee opened 1 month ago

Erin-Chee commented 1 month ago

Operating System

CentOS 7

Other Linux

No response

Workflow Version

v1.2.0-g36f41e7

Workflow Execution

Command line (Cluster)

Other workflow execution

No response

EPI2ME Version

No response

CLI command run

!/bin/bash

SBATCH -p defq

SBATCH -c 2

export SINGULARITY_BIND="/mnt/storage/nobackup/proj/scbsu/nec181/testing/EPI2ME/wf-alignment/" export SINGULARITYENV_PREPEND_PATH="/home/epi2melabs/conda/bin/"

module load Java/17.0.6

nextflow run epi2me-labs/wf-alignment \ --fastq wf-alignment-demo/fastq \ --references wf-alignment-demo/references \ -profile singularity

Workflow Execution - CLI Execution Profile

singularity

What happened?

Hi, I've been encountering errors related to container usage (I think) when trying to run wf-alignment with the demo data.

To try and diagnose the issue, I ran singularity shell ontresearch-wf-alignment-shaa9faef16822c5aa48366a4c45b401c9233a6c0f7.img python --version but it returned bash: python: command not found

I had gotten a similar error when running wf-human-variation but resolved it by setting: export SINGULARITYENV_PREPEND_PATH="/home/epi2melabs/conda/bin/" before starting the pipeline. However, this solution hasn't worked in this case.

I also tried starting an interactive shell session with ontresearch-wf-common-sha338caea0a2532dc0ea8f46638ccc322bb8f9af48.img but encountered the following error: ERROR: /bin/sh does not exist in container

How should I fix this? Any help is much appreciated. Thanks!

Relevant log output

N E X T F L O W   ~  version 24.04.3

Launching `https://github.com/epi2me-labs/wf-alignment` [compassionate_brattain] DSL2 - revision: 36f41e7966 [master]

||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-alignment v1.2.0-g36f41e7
--------------------------------------------------------------------------------
Core Nextflow options
  revision       : master
  runName        : compassionate_brattain
  containerEngine: singularity
  container      : [withLabel:wfalignment:/mnt/storage/nobackup/proj/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity/ontresearch-wf-alignment/, withLabel:wf_common:/mnt/storage/nobackup/proj/scb
su/nec181/testing/EPI2ME/wf-alignment/work/singularity/ontresearch-wf-common/]
  launchDir      : /mnt/storage/nobackup/proj/scbsu/nec181/testing/EPI2ME/wf-alignment
  workDir        : /mnt/storage/nobackup/proj/scbsu/nec181/testing/EPI2ME/wf-alignment/work
  projectDir     : /mnt/nfs/home/nec181/.nextflow/assets/epi2me-labs/wf-alignment
  userName       : nec181
  profile        : singularity
  configFiles    : /mnt/nfs/home/nec181/.nextflow/assets/epi2me-labs/wf-alignment/nextflow.config

Input Options
  fastq          : wf-alignment-demo/fastq
  references     : wf-alignment-demo/references

!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-alignment for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

--------------------------------------------------------------------------------
This is epi2me-labs/wf-alignment v1.2.0-g36f41e7.
--------------------------------------------------------------------------------
Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
[-        ] fastcat -

[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       -
[-        ] pipeline:getParams             -
[-        ] pipeline:getVersions           -
[-        ] pip…:process_references:fx2tab -
[-        ] pip…process_references:combine -
[-        ] pipeline:makeMMIndex           -

[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       -
[-        ] pipeline:getParams             -
[-        ] pipeline:getVersions           -
[-        ] pip…:process_references:fx2tab -
[-        ] pip…process_references:combine -
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -

[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       -
[-        ] pipeline:getParams             -
[-        ] pipeline:getVersions           -
[-        ] pip…:process_references:fx2tab -
[-        ] pip…process_references:combine -
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         -
[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       -
[-        ] pipeline:getParams             -
[-        ] pipeline:getVersions           -
[-        ] pip…:process_references:fx2tab -
[-        ] pip…process_references:combine -
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -

[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       -
[-        ] pipeline:getParams             -
[-        ] pipeline:getVersions           -
[-        ] pip…:process_references:fx2tab -
[-        ] pip…process_references:combine -
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         -

[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       | 0 of 3
[-        ] pipeline:getParams             | 0 of 1
[-        ] pipeline:getVersions           | 0 of 1
[-        ] pip…:process_references:fx2tab | 0 of 3
[-        ] pip…process_references:combine | 0 of 1
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         -

executor >  local (2)
[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       | 0 of 3
[e2/51519b] pipeline:getParams             | 0 of 1
[ab/fcc7a1] pipeline:getVersions           | 0 of 1
[-        ] pip…:process_references:fx2tab | 0 of 3
[-        ] pip…process_references:combine | 0 of 1
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         -

executor >  local (2)
[-        ] fastcat                        -
[-        ] move_or_compress_fq_file       | 0 of 3
[e2/51519b] pipeline:getParams             | 0 of 1
[ab/fcc7a1] pipeline:getVersions           | 0 of 1
[-        ] pip…:process_references:fx2tab | 0 of 3
[-        ] pip…process_references:combine | 0 of 1
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         -

executor >  local (4)
[-        ] fastcat                        -
[fb/ad59ce] move_or_compress_fq_file (1)   | 0 of 3
[e2/51519b] pipeline:getParams             | 1 of 1 ✔
[ab/fcc7a1] pipeline:getVersions           | 0 of 1
[df/1aaa27] pip…cess_references:fx2tab (2) | 0 of 3
[-        ] pip…process_references:combine | 0 of 1
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         | 0 of 1
ERROR ~ Error executing process > 'pipeline:getVersions'

Caused by:
  Process `pipeline:getVersions` terminated with an error exit status (127)

Command executed:

  python --version | tr -s ' ' ',' | tr '[:upper:]' '[:lower:]' > versions.txt
  seqkit version | sed 's/ /,/' >> versions.txt
  minimap2 --version | sed 's/^/minimap2,/' >> versions.txt
  samtools --version | (head -n 1 && exit 0) | sed 's/ /,/' >> versions.txt
  fastcat --version | sed 's/^/fastcat,/' >> versions.txt
  mosdepth --version | sed 's/ /,/' >> versions.txt
  ezcharts --version | sed 's/ /,/' >> versions.txt
  python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
  bgzip --version | head -n1 | sed -E 's/(.*) /\1,/' >> versions.txt

Command exit status:
  127

Command output:
  (empty)

Command error:
  WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  .command.sh: line 2: python: command not found

Work dir:
  /mnt/storage/nobackup/proj/scbsu/nec181/testing/EPI2ME/wf-alignment/work/ab/fcc7a164d38ce8b15b4a998434231c

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
WARN: Killing running tasks (2)

executor >  local (4)
[-        ] fastcat                        -
[fb/ad59ce] move_or_compress_fq_file (1)   | 0 of 2
[e2/51519b] pipeline:getParams             | 1 of 1 ✔
[ab/fcc7a1] pipeline:getVersions           | 1 of 1, failed: 1 ✘
[df/1aaa27] pip…cess_references:fx2tab (2) | 0 of 2
[-        ] pip…process_references:combine | 0 of 1
[-        ] pipeline:makeMMIndex           -
[-        ] pipeline:alignReads            -
[-        ] pipeline:bamstats              -
[-        ] pipeline:addStepsColumn        -
[-        ] pipeline:readDepthPerRef       -
[-        ] pipeline:collectFilesInDir     -
[-        ] pipeline:makeReport            -
[-        ] pipeline:configure_igv         -
[-        ] output                         | 0 of 1
ERROR ~ Error executing process > 'pipeline:getVersions'

Caused by:
  Process `pipeline:getVersions` terminated with an error exit status (127)

Command executed:

  python --version | tr -s ' ' ',' | tr '[:upper:]' '[:lower:]' > versions.txt
  seqkit version | sed 's/ /,/' >> versions.txt
  minimap2 --version | sed 's/^/minimap2,/' >> versions.txt
  samtools --version | (head -n 1 && exit 0) | sed 's/ /,/' >> versions.txt
  fastcat --version | sed 's/^/fastcat,/' >> versions.txt
  mosdepth --version | sed 's/ /,/' >> versions.txt
  ezcharts --version | sed 's/ /,/' >> versions.txt
  python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
  bgzip --version | head -n1 | sed -E 's/(.*) /\1,/' >> versions.txt

Command exit status:
  127

Command output:
  (empty)

Command error:
  WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  .command.sh: line 2: python: command not found

Work dir:
  /mnt/storage/nobackup/proj/scbsu/nec181/testing/EPI2ME/wf-alignment/work/ab/fcc7a164d38ce8b15b4a998434231c

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

no

Other demo data information

No response

SamStudio8 commented 1 month ago

@Erin-Chee Are you able to provide a debug output for your command to find python?

singularity --debug exec ontresearch-wf-alignment-shaa9faef16822c5aa48366a4c45b401c9233a6c0f7.img which python
SamStudio8 commented 1 month ago

@Erin-Chee Actually, I should have asked more simply, does this work if you provide --no-home?

singularity exec --no-home ontresearch-wf-alignment-shaa9faef16822c5aa48366a4c45b401c9233a6c0f7.img which python
Erin-Chee commented 1 month ago

Hi @SamStudio8 thanks for the reply.

I ran singularity exec with --no-home, but there was no output for which python.

I tried it with --debug and this was the output:

DEBUG [U=599395,P=11349] persistentPreRun() Singularity version: 3.11.5-1.el7 DEBUG [U=599395,P=11349] persistentPreRun() Parsing configuration file /etc/singularity/singularity.conf DEBUG [U=599395,P=11349] handleConfDir() /mnt/nfs/home/nec181/.singularity already exists. Not creating. DEBUG [U=599395,P=11349] launchContainer() Using native runtime launcher. DEBUG [U=599395,P=11349] setUmask() Saving umask 0077 for propagation into container DEBUG [U=599395,P=11349] Init() Image format detection DEBUG [U=599395,P=11349] Init() Check for sandbox image format DEBUG [U=599395,P=11349] Init() sandbox format initializer returned: not a directory image DEBUG [U=599395,P=11349] Init() Check for sif image format DEBUG [U=599395,P=11349] Init() sif image format detected DEBUG [U=599395,P=11349] checkEncryptionKey() Checking for encrypted system partition DEBUG [U=599395,P=11349] setCgroups() Recording rootless XDG_RUNTIME_DIR / DBUS_SESSION_BUS_ADDRESS DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_PAPER environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding MANPATH environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding XDG_SESSION_ID environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_ADDRESS environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding HOSTNAME environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_MONETARY environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding TERM environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding SHELL environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_ROOT environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding HISTSIZE environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding MODULEPATH_ROOT environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding TMPDIR environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding SSH_CLIENT environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_PKG environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding I_MPI_PMI_LIBRARY environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_NUMERIC environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding OLDPWD environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_VERSION environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding SSH_TTY environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding USER environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_sys environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LS_COLORS environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_TELEPHONE environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding TMOUT environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding MAIL environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_IDENTIFICATION environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding PWD environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LANG environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding MODULEPATH environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_MEASUREMENT environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_CMD environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding HISTCONTROL environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding SHLVL environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding BASH_ENV environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LOGNAME environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding SSH_CONNECTION environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding MODULESHOME environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_SETTARG_FULL_SUPPORT environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LESSOPEN environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding XDG_RUNTIME_DIR environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LMOD_DIR environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_TIME environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding LC_NAME environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding BASH_FUNC_module() environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding BASH_FUNCml() environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding environment variable VERBOSE [U=599395,P=11349] SetContainerEnv() Not forwarding SINGULARITY_DEBUG environment variable DEBUG [U=599395,P=11349] SetContainerEnv() Forwarding USER_PATH environment variable VERBOSE [U=599395,P=11349] SetContainerEnv() Setting HOME=/mnt/nfs/home/nec181 VERBOSE [U=599395,P=11349] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DEBUG [U=599395,P=11349] init() Use starter binary /usr/libexec/singularity/bin/starter-suid DEBUG [U=599395,P=11349] Exec() Setting GOGC=off for starter VERBOSE [U=0,P=11349] print() Set messagelevel to: 5 VERBOSE [U=0,P=11349] init() Starter initialization DEBUG [U=0,P=11349] load_overlay_module() Trying to load overlay kernel module DEBUG [U=0,P=11349] load_overlay_module() Overlay seems supported by the kernel VERBOSE [U=0,P=11349] is_suid() Check if we are running as setuid VERBOSE [U=0,P=11349] priv_drop() Drop root privileges DEBUG [U=599395,P=11349] read_engine_config() Read engine configuration DEBUG [U=599395,P=11349] init() CleanupHost not requested DEBUG [U=599395,P=11349] cleanenv() Clearing env var LOAD_OVERLAY_MODULE=1 DEBUG [U=599395,P=11349] cleanenv() Keeping env var SINGULARITY_MESSAGELEVEL=5 DEBUG [U=599395,P=11349] cleanenv() Clearing env var ENGINE_CONFIG1={"engineName":"singularity","containerID":"","engineConfig":{"jsonConfig":{"unixSocketPair":[0,0],"image":"/mnt/nfs/home/nec181/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity/ontresearch-wf-alignment-shaa9faef16822c5aa48366a4c45b401c9233a6c0f7.img","homedir":"/mnt/nfs/home/nec181","homeDest":"/mnt/nfs/home/nec181","network":"bridge","cwd":"/mnt/nfs/home/nec181/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity","configurationFile":"/etc/singu(TRUNCATED...)DEBUG [U=599395,P=11349] cleanenv() Clearing env var ENGINE_CONFIG_CHUNKS=1 DEBUG [U=599395,P=11349] cleanenv() Keeping env var GOGC=off DEBUG [U=599395,P=11349] init() Wait completion of stage1 VERBOSE [U=599395,P=11364] priv_drop() Drop root privileges permanently DEBUG [U=599395,P=11364] set_parent_death_signal() Set parent death signal to 9 VERBOSE [U=599395,P=11364] init() Spawn stage 1 DEBUG [U=599395,P=11364] startup() singularity runtime engine selected VERBOSE [U=599395,P=11364] startup() Execute stage 1 DEBUG [U=599395,P=11364] StageOne() Entering stage 1 DEBUG [U=599395,P=11364] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point DEBUG [U=599395,P=11364] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point DEBUG [U=599395,P=11364] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point DEBUG [U=599395,P=11364] prepareAutofs() Could not keep file descriptor for home directory /mnt/nfs/home/nec181: no mount point DEBUG [U=599395,P=11364] prepareAutofs() Could not keep file descriptor for current working directory /mnt/nfs/home/nec181/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity: no mount point DEBUG [U=599395,P=11364] Init() Image format detection DEBUG [U=599395,P=11364] Init() Check for sandbox image format DEBUG [U=599395,P=11364] Init() sandbox format initializer returned: not a directory image DEBUG [U=599395,P=11364] Init() Check for sif image format DEBUG [U=599395,P=11364] Init() sif image format detected DEBUG [U=599395,P=11364] setSessionLayer() Overlay seems supported and allowed by kernel DEBUG [U=599395,P=11364] setSessionLayer() Attempting to use overlayfs (enable overlay = try) VERBOSE [U=599395,P=11349] wait_child() stage 1 exited with status 0 DEBUG [U=599395,P=11349] cleanup_fd() Setting FD_CLOEXEC on starter fd 3 DEBUG [U=599395,P=11349] cleanup_fd() Close file descriptor 4 DEBUG [U=599395,P=11349] cleanup_fd() Close file descriptor 5 DEBUG [U=599395,P=11349] cleanup_fd() Close file descriptor 6 DEBUG [U=599395,P=11349] cleanup_fd() Close file descriptor 7 DEBUG [U=599395,P=11349] cleanup_fd() Close file descriptor 8 DEBUG [U=599395,P=11349] init() Set child signal mask DEBUG [U=599395,P=11349] init() Create socketpair for master communication channel DEBUG [U=599395,P=11349] init() Create RPC socketpair for communication between stage 2 and RPC server VERBOSE [U=599395,P=11349] priv_escalate() Get root privileges VERBOSE [U=0,P=11349] priv_escalate() Change filesystem uid to 599395 VERBOSE [U=0,P=11349] init() Spawn master process DEBUG [U=0,P=11370] set_parent_death_signal() Set parent death signal to 9 VERBOSE [U=0,P=11370] create_namespace() Create mount namespace VERBOSE [U=0,P=11349] enter_namespace() Entering in mount namespace DEBUG [U=0,P=11349] enter_namespace() Opening namespace file ns/mnt VERBOSE [U=0,P=11370] create_namespace() Create mount namespace DEBUG [U=0,P=11349] set_master_privileges() Set master privileges DEBUG [U=0,P=11349] apply_privileges() Effective capabilities: 0x00000000000000c0 DEBUG [U=0,P=11349] apply_privileges() Permitted capabilities: 0x0000001fffffffff DEBUG [U=0,P=11349] apply_privileges() Bounding capabilities: 0x0000001fffffffff DEBUG [U=0,P=11349] apply_privileges() Inheritable capabilities: 0x0000001fffffffff DEBUG [U=0,P=11349] apply_privileges() Ambient capabilities: 0x0000000000000000 DEBUG [U=0,P=11349] apply_privileges() Set user ID to 599395 DEBUG [U=0,P=11371] set_rpc_privileges() Set RPC privileges DEBUG [U=0,P=11371] apply_privileges() Effective capabilities: 0x0000000000200000 DEBUG [U=0,P=11371] apply_privileges() Permitted capabilities: 0x0000001fffffffff DEBUG [U=0,P=11371] apply_privileges() Bounding capabilities: 0x0000000008204000 DEBUG [U=0,P=11371] apply_privileges() Inheritable capabilities: 0x0000000000000000 DEBUG [U=0,P=11371] apply_privileges() Ambient capabilities: 0x0000000000000000 DEBUG [U=0,P=11371] apply_privileges() Set user ID to 599395 DEBUG [U=599395,P=11371] set_parent_death_signal() Set parent death signal to 9 VERBOSE [U=599395,P=11371] init() Spawn RPC server DEBUG [U=599395,P=11349] startup() singularity runtime engine selected VERBOSE [U=599395,P=11349] startup() Execute master process DEBUG [U=599395,P=11371] startup() singularity runtime engine selected VERBOSE [U=599395,P=11371] startup() Serve RPC requests DEBUG [U=599395,P=11349] setupSessionLayout() Using Layer system: overlay DEBUG [U=599395,P=11349] setupOverlayLayout() Creating overlay SESSIONDIR layout DEBUG [U=599395,P=11349] addRootfsMount() Mount rootfs in read-only mode DEBUG [U=599395,P=11349] addRootfsMount() Image type is 4096 DEBUG [U=599395,P=11349] addRootfsMount() Mounting block [squashfs] image: /mnt/nfs/home/nec181/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity/ontresearch-wf-alignment-shaa9faef16822c5aa48366a4c45b401c9233a6c0f7.img DEBUG [U=599395,P=11349] addKernelMount() Checking configuration file for 'mount proc' DEBUG [U=599395,P=11349] addKernelMount() Adding proc to mount list VERBOSE [U=599395,P=11349] addKernelMount() Default mount: /proc:/proc DEBUG [U=599395,P=11349] addKernelMount() Checking configuration file for 'mount sys' DEBUG [U=599395,P=11349] addKernelMount() Adding sysfs to mount list VERBOSE [U=599395,P=11349] addKernelMount() Default mount: /sys:/sys DEBUG [U=599395,P=11349] addDevMount() Checking configuration file for 'mount dev' DEBUG [U=599395,P=11349] addDevMount() Adding dev to mount list VERBOSE [U=599395,P=11349] addDevMount() Default mount: /dev:/dev DEBUG [U=599395,P=11349] addHostMount() Not mounting host file systems per configuration VERBOSE [U=599395,P=11349] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime VERBOSE [U=599395,P=11349] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts DEBUG [U=599395,P=11349] addHomeStagingDir() Staging home directory (/mnt/nfs/home/nec181) at /var/lib/singularity/mnt/session/mnt/nfs/home/nec181 DEBUG [U=599395,P=11349] addHomeMount() Adding home directory mount [/var/lib/singularity/mnt/session/mnt/nfs/home/nec181:/mnt/nfs/home/nec181] to list using layer: overlay DEBUG [U=599395,P=11349] addTmpMount() Checking for 'mount tmp' in configuration file VERBOSE [U=599395,P=11349] addTmpMount() Default mount: /tmp:/tmp VERBOSE [U=599395,P=11349] addTmpMount() Default mount: /var/tmp:/var/tmp DEBUG [U=599395,P=11349] addScratchMount() Not mounting scratch directory: Not requested DEBUG [U=599395,P=11349] addLibsMount() Checking for 'user bind control' in configuration file DEBUG [U=599395,P=11349] addFilesMount() Checking for 'user bind control' in configuration file DEBUG [U=599395,P=11349] addResolvConfMount() Adding /etc/resolv.conf to mount list VERBOSE [U=599395,P=11349] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf DEBUG [U=599395,P=11349] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request DEBUG [U=599395,P=11349] create() Mount all DEBUG [U=599395,P=11349] mountGeneric() Mounting tmpfs to /var/lib/singularity/mnt/session DEBUG [U=599395,P=11349] mountImage() Mounting loop device /dev/loop0 to /var/lib/singularity/mnt/session/rootfs of type squashfs DEBUG [U=599395,P=11349] mountGeneric() Mounting overlay to /var/lib/singularity/mnt/session/final DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/final DEBUG [U=599395,P=11349] setPropagationMount() Set RPC mount propagation flag to SLAVE VERBOSE [U=599395,P=11349] Passwd() Checking for template passwd file: /var/lib/singularity/mnt/session/rootfs/etc/passwd WARNING [U=599395,P=11349] addIdentityMount() passwd file doesn't exist in container, not updating VERBOSE [U=599395,P=11349] Group() Checking for template group file: /var/lib/singularity/mnt/session/rootfs/etc/group WARNING [U=599395,P=11349] addIdentityMount() group file doesn't exist in container, not updating DEBUG [U=599395,P=11349] mountGeneric() Mounting /dev to /var/lib/singularity/mnt/session/final/dev DEBUG [U=599395,P=11349] mountGeneric() Mounting /etc/localtime to /var/lib/singularity/mnt/session/final/etc/localtime DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/final/etc/localtime DEBUG [U=599395,P=11349] mountGeneric() Mounting /etc/hosts to /var/lib/singularity/mnt/session/final/etc/hosts DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/final/etc/hosts DEBUG [U=599395,P=11349] mountGeneric() Mounting /proc to /var/lib/singularity/mnt/session/final/proc DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/final/proc DEBUG [U=599395,P=11349] mountGeneric() Mounting sysfs to /var/lib/singularity/mnt/session/final/sys DEBUG [U=599395,P=11349] mountGeneric() Mounting /mnt/nfs/home/nec181 to /var/lib/singularity/mnt/session/mnt/nfs/home/nec181 DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/mnt/nfs/home/nec181 DEBUG [U=599395,P=11349] mountGeneric() Mounting /var/lib/singularity/mnt/session/mnt/nfs/home/nec181 to /var/lib/singularity/mnt/session/final/mnt/nfs/home/nec181 DEBUG [U=599395,P=11349] mountGeneric() Mounting /tmp to /var/lib/singularity/mnt/session/final/tmp DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/final/tmp DEBUG [U=599395,P=11349] mountGeneric() Mounting /var/tmp to /var/lib/singularity/mnt/session/final/var/tmp DEBUG [U=599395,P=11349] mountGeneric() Remounting /var/lib/singularity/mnt/session/final/var/tmp DEBUG [U=599395,P=11349] mountGeneric() Mounting /var/lib/singularity/mnt/session/etc/resolv.conf to /var/lib/singularity/mnt/session/final/etc/resolv.conf DEBUG [U=599395,P=11349] addCwdMount() Using /mnt/nfs/home/nec181/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity as current working directory VERBOSE [U=599395,P=11349] addCwdMount() /mnt/nfs/home/nec181/scbsu/nec181/testing/EPI2ME/wf-alignment/work/singularity found within container DEBUG [U=599395,P=11349] create() Chroot into /var/lib/singularity/mnt/session/final DEBUG [U=599395,P=11371] Chroot() Hold reference to host / directory DEBUG [U=599395,P=11371] Chroot() Called pivot_root on /var/lib/singularity/mnt/session/final DEBUG [U=599395,P=11371] Chroot() Change current directory to host / directory DEBUG [U=599395,P=11371] Chroot() Apply slave mount propagation for host / directory DEBUG [U=599395,P=11371] Chroot() Called unmount(/, syscall.MNT_DETACH) DEBUG [U=599395,P=11371] Chroot() Changing directory to / to avoid getpwd issues DEBUG [U=599395,P=11349] create() Chdir into / to avoid errors VERBOSE [U=0,P=11370] wait_child() rpc server exited with status 0 DEBUG [U=0,P=11370] init() Set container privileges DEBUG [U=0,P=11370] apply_privileges() Effective capabilities: 0x0000000000000000 DEBUG [U=0,P=11370] apply_privileges() Permitted capabilities: 0x0000000000000000 DEBUG [U=0,P=11370] apply_privileges() Bounding capabilities: 0x0000000000000000 DEBUG [U=0,P=11370] apply_privileges() Inheritable capabilities: 0x0000000000000000 DEBUG [U=0,P=11370] apply_privileges() Ambient capabilities: 0x0000000000000000 DEBUG [U=0,P=11370] apply_privileges() Set user ID to 599395 DEBUG [U=599395,P=11370] set_parent_death_signal() Set parent death signal to 9 DEBUG [U=599395,P=11370] startup() singularity runtime engine selected VERBOSE [U=599395,P=11370] startup() Execute stage 2 DEBUG [U=599395,P=11370] StageTwo() Entering stage 2 DEBUG [U=599395,P=11370] StartProcess() Setting umask in container to 0077 DEBUG [U=599395,P=11370] func1() Not exporting "BASH_FUNC_ml()" to container environment: invalid key DEBUG [U=599395,P=11370] func1() Not exporting "BASH_FUNC_module()" to container environment: invalid key DEBUG [U=599395,P=11370] sylogBuiltin() Running action command exec DEBUG [U=599395,P=11349] PostStartProcess() Post start process DEBUG [U=599395,P=11349] Master() Child exited with exit status 1