search

nf-core / eager

A fully reproducible and state-of-the-art ancient DNA analysis pipeline
https://nf-co.re/eager
MIT License
146 stars 82 forks source link

Singularity missing? #68

Closed apeltzer closed 5 years ago

apeltzer commented 5 years ago 
> nextflow run nf-core/eager -with-singularity /apps/containers/nf-core-eager.simg  --pairedEnd --reads "/path/sample_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome ${ref} --saveReference true  -name textEager3
N E X T F L O W  ~  version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: 897fca777a [master]
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
=========================================
 nf-core/eager v2.0.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.0.0
Run Name       : textEager3
Reads          : /path/sample_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/user/eager/work
Container Engine: singularity
Container      : /apps/containers/nf-core-eager.simg
Current home   : /home/user
Current user   : user
Current path   : /home/user/fastdir/eager
Script dir     : /home/user/.nextflow/assets/nf-core/eager
Config Profile : standard
=========================================
[warm up] executor > SLURM
[c5/898ed5] Submitted process > get_software_versions
[e1/cadfbc] Submitted process > fastqc (sample)
[06/d3a688] Submitted process > adapter_removal (sample)
[7f/7a6a9a] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Process `get_software_versions` terminated with an error exit status (127)

Command executed:

  echo 2.0.0 &> v_pipeline.txt
  echo 0.32.0 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  bwa &> v_bwa.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  AdapterRemoval -version  &> v_adapterremoval.txt 2>&1 || true
  picard MarkDuplicates --version &> v_markduplicates.txt  2>&1 || true
  dedup -v &> v_dedup.txt 2>&1 || true
  preseq &> v_preseq.txt 2>&1 || true
  gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
  vcf2genome &> v_vcf2genome.txt 2>&1 || true
  fastp --version &> v_fastp.txt 2>&1 || true
  bam --version &> v_bamutil.txt 2>&1 || true
  qualimap --version &> v_qualimap.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  127

Command output:
  (empty)

Command error:
  env: singularity: No such file or directory

Work dir:
  /fast/users/user/eager/work/c5/898ed52fe4a02f48caaf2f0130963b

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[e1/cadfbc] NOTE: Process `fastqc (sample)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete

Originally posted by @yassineS in https://github.com/nf-core/eager/issues/67#issuecomment-434595084

apeltzer commented 5 years ago

Can you confirm that singularity is installed properly? 


Command exit status:
  127

Command output:
  (empty)

Command error:
  env: singularity: No such file or directory

This suggests its not installed on your cluster and Nextflow can't call singularity properly. Maybe some missing module activation? You can by the way also simply run like this:

-profile singularity

without requiring the -with-Singularity option at all. The image will then be pulled automatically and stored in the work directory of the current run.

yassineS commented 5 years ago

I have rerun with the changes suggested above and on the other thread, I still get the 127 error:

N E X T F L O W  ~  version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: 4ac6d7a330 [dev]
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
=========================================
 nf-core/eager v2.0.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name       : textEager3
Reads          : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/a1222423/eager/work
Container Engine: singularity
Container      : shub://nf-core/eager:latest
Current home   : /home/a1222423
Current user   : a1222423
Current path   : /home/a1222423/fastdir/eager
Script dir     : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : singularity
=========================================
[warm up] executor > SLURM
[da/19ef48] Submitted process > fastqc (NGSD2_r2)
[4b/c2f569] Submitted process > get_software_versions
[38/a1632e] Submitted process > adapter_removal (NGSD2_r2)
[10/71f24c] Submitted process > output_documentation
ERROR ~ Error executing process > 'fastqc (NGSD2_r2)'

Caused by:
  Process `fastqc (NGSD2_r2)` terminated with an error exit status (127)

Command executed:

  fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /fast/users/a1222423/eager/work/da/19ef48c42b7eda36c089592a5536f6: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /fast/users/a1222423/eager/work/da/19ef48c42b7eda36c089592a5536f6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
[nf-core/eager] Pipeline Complete
apeltzer commented 5 years ago

Is singularity present on that machine?

yassineS commented 5 years ago

Yes!

$ singularity --version
2.5.2-dist
apeltzer commented 5 years ago

Okay, it looks a bit weird because the error here:


  Process `fastqc (NGSD2_r2)` terminated with an error exit status (127)

Command executed:

  fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz

Command exit status:
  127

Suggests via exit code = 127 that fastqc can't be found.

Also, this here:

Config Profile : singularity
=========================================
[warm up] executor > SLURM

states that you use a SLURM scheduler but didn't specify a profile for this. Do you use your own local configuration to submit jobs to a scheduler? In case so, I can add a configuration for your local hpc environment that should load required modules e.g. singularity automatically, as it could be the case it doesn't load singularity on cluster nodes right now.

yassineS commented 5 years ago

There’s no special configuration required to launch a slurm job on our HPC. Does NF require any special configuration?

Y

Cheers, Yassine Souilmi

From: Alexander Peltzer notifications@github.com Sent: Friday, November 2, 2018 11:53:19 PM To: nf-core/eager Cc: Yassine Souilmi; Mention Subject: Re: [nf-core/eager] Singularity missing? (#68)

Okay, it looks a bit weird because the error here:

Process fastqc (NGSD2_r2) terminated with an error exit status (127)

Command executed:

fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz

Command exit status: 127

Suggests via exit code = 127 that fastqc can't be found.

Also, this here:

Config Profile : singularity

[warm up] executor > SLURM

states that you use a SLURM scheduler but didn't specify a profile for this. Do you use your own local configuration to submit jobs to a scheduler? In case so, I can add a configuration for your local hpc environment that should load required modules e.g. singularity automatically, as it could be the case it doesn't load singularity on cluster nodes right now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/nf-core/eager/issues/68#issuecomment-435378698, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABr4IVBd7-pBmQR2c2S2Ptr9aIBQ3HL6ks5urEdHgaJpZM4YDlNE.

apeltzer commented 5 years ago

Depends on the cluster quite a bit! Your error message could be explained by submitting jobs to slurm / cluster nodes, but without loading a Singularity module there and thus the tools inside the container cannot be resolved/used (error 127).

How do you ensure that Singularity is up and running on the cluster? Do you load a module or something similar?

yassineS commented 5 years ago

Yes! I load the module and double check that it’s callable. Never faced an issue where a module needs to be loaded on the compute nodes.

I have been doing some digging, and the problem might be similar to (https://github.com/nf-core/rnaseq/issues/29#issuecomment-398706617 https://github.com/nf-core/rnaseq/issues/29#issuecomment-398706617). But I am not really sure.

Thanks for your help.

On 4 Nov 2018, at 8:27 pm, Alexander Peltzer <notifications@github.com mailto:notifications@github.com> wrote:

Depends on the cluster quite a bit! Your error message could be explained by submitting jobs to slurm / cluster nodes, but without loading a Singularity module there and thus the tools inside the container cannot be resolved/used (error 127).

How do you ensure that Singularity is up and running on the cluster? Do you load a module or something similar?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nf-core/eager/issues/68#issuecomment-435656270, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr4IVthTf9vtO6PrNkNmGvO8eWjJqCLks5urrnxgaJpZM4YDlNE.

apeltzer commented 5 years ago

Hey! Can you let me know how you load the module and maybe just give me a name for the cluster? I can then come up with a -profile <your_cluster_name> that should run :+1: 

module load modules/singularity/2.5.2
yassineS commented 5 years ago

The cluster is called Phoenix, and I load the module using the following command:

module load Singularity/2.5.2-GCC-5.4.0-2.26

On 4 Nov 2018, at 8:33 pm, Alexander Peltzer <notifications@github.com mailto:notifications@github.com> wrote:

Hey! Can you let me know how you load the module and maybe just give me a name for the cluster? I can then come up with a -profile that should run 👍

module load modules/singularity/2.5.2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nf-core/eager/issues/68#issuecomment-435656666, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr4ITNGP7cNkVm8gX1dw-PxLr51TWF8ks5urruBgaJpZM4YDlNE.

apeltzer commented 5 years ago

Thanks a bunch!

I added a config profile for you to test - its called -profile phoenix and is currently only found in the dev branch, therefore run your command using this:

nextflow run [your previous options] -profile phoenix -r dev
yassineS commented 5 years ago

Unfortunately, this didn't work. It first returned the following error:

WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /fast/users/a1222423/eager/work/singularity
Pulling Singularity image docker://nfcore/eager:latest [cache /fast/users/a1222423/eager/work/singularity/nfcore-eager-latest.img]
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Failed to pull singularity image
  command: singularity pull --name nfcore-eager-latest.img docker://nfcore/eager:latest > /dev/null
  status : 255
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
    WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
    ERROR: You must install squashfs-tools to build images
    ABORT: Aborting with RETVAL=255
    ERROR: pulling container failed!

 -- Check '.nextflow.log' file for details
[nf-core/eager] Pipeline Complete

Which I tried to circumvent by downloading the image manually, and than rerunning the workflow:

$ singularity pull --name nfcore-eager.img shub://nf-core/eager:latest
Progress |===================================| 100.0%
Done. Container is at: /fast/users/a1222423/eager/nfcore-eager.img
$
$
$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true  -profile phoenix -with-singularity nfcore-eager.img  -name textEager3
N E X T F L O W  ~  version 0.32.0
Project `nf-core/eager` currently is sticked on revision: dev -- you need to specify explicitly a revision with the option `-r` to use it
[a1222423@l03 eager]$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true  -profile phoenix -with-singularity nfcore-eager.img -r dev -name textEager3
N E X T F L O W  ~  version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: a6bc45406e [dev]
=========================================
 nf-core/eager v2.1.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name       : textEager3
Reads          : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/a1222423/eager/work
Container Engine: singularity
Container      : nfcore-eager.img
Current home   : /home/a1222423
Current user   : a1222423
Current path   : /home/a1222423/fastdir/eager
Script dir     : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : phoenix
=========================================
[warm up] executor > slurm
[2b/b9706a] Submitted process > get_software_versions
[8e/3ae677] Submitted process > fastqc (NGSD2_r2)
[68/3dd262] Submitted process > adapter_removal (NGSD2_r2)
[c1/03664f] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Process `get_software_versions` terminated with an error exit status (127)

Command executed:

  echo 2.1.0dev &> v_pipeline.txt
  echo 0.32.0 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  bwa &> v_bwa.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  AdapterRemoval -version  &> v_adapterremoval.txt 2>&1 || true
  picard MarkDuplicates --version &> v_markduplicates.txt  2>&1 || true
  dedup -v &> v_dedup.txt 2>&1 || true
  preseq &> v_preseq.txt 2>&1 || true
  gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
  vcf2genome &> v_vcf2genome.txt 2>&1 || true
  fastp --version &> v_fastp.txt 2>&1 || true
  bam --version &> v_bamutil.txt 2>&1 || true
  qualimap --version &> v_qualimap.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /fast/users/a1222423/eager/work/2b/b9706a04822a46b640fa5fc67d10c6: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /fast/users/a1222423/eager/work/2b/b9706a04822a46b640fa5fc67d10c6

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[8e/3ae677] NOTE: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete
apeltzer commented 5 years ago

Ok, the first issue can be resolved by installing squashfs-tools on the server once. Maybe ask IT to do so, we had to as well ;-)

The second issue seems indeed to be related to the link you posted: https://github.com/nf-core/rnaseq/issues/29#issuecomment-398706617

It looks like your cluster admin (?) didn't specify certain configuration parameters for singularity to access certain paths on your system. For example, we have a system where we can only run images and access files stored under /beegfs/<....> and this has to be configured in the Singularity installation once for all users.

I can help you in getting that up and communicate it appropriately if you want to, but we'd need to have a shared screen session / Skype to talk about this ...

yassineS commented 5 years ago

Sounds good to me, and thank you again for your support!

apeltzer commented 5 years ago

No worries, happy to have some people testing the pipeline and giving feedback like this - thanks for testing and send me a DM/email if we should do a screen share or similar session to figure out whats going on.

apeltzer commented 5 years ago

Heads up to document for future cases what goes on:

Head node uses a different home than worker nodes on cluster, this leads to entire work directories not being readable, thus the error of missing things. Had similar issues on another cluster before, where the homes were linked as

/home-link/<yourusername> and then on the node they were present as /home/<yourusername>...

uoabowen commented 5 years ago

Hi @apeltzer,

I am the guy who installed and configured singularity on Phoenix which @yassineS is using.

We use $SINGULARITY_BINDPATH to bind directory, see https://www.sylabs.io/guides/2.6/user-guide/bind_paths_and_mounts.html#binding-with-overlay for details

This is my test script

/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ echo $SINGULARITY_BINDPATH 
/fast/users/a1222423/eager/work,/data
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ cat test-bowen.sh 
#!/bin/bash -x
#SBATCH -p test
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --mem=1G
#SBATCH --time=15:00

ml Singularity/2.5.2-GCC-5.4.0-2.26

export PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:$PATH

singularity exec /fast/users/a1222423/eager/work/singularity/nf-core-eager.img /bin/bash -c "pwd; cd $PWD; pwd; ls -al; /bin/bash /fast/users/a1222423/eager/work/8b//12a213d46743cf1ccdd3e71e23f23f/.command.stub"

This is what I got

[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ cat slurm-12963871.out
+ LMOD_PKG=/usr/share/lmod/lmod
+ LMOD_DIR=/usr/share/lmod/lmod/libexec
+ LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod
+ MODULESHOME=/usr/share/lmod/lmod
+ export LMOD_PKG
+ export LMOD_CMD
+ export LMOD_DIR
+ export MODULESHOME
+ LMOD_VERSION=6.3.4
+ export LMOD_VERSION
+ '[' : '!=' : ']'
+ unalias ml
+ true
++ /usr/bin/tr '[:upper:]' '[:lower:]'
++ echo YES
+ export_module=yes
+ '[' -n '4.2.46(2)-release' -a yes '!=' no ']'
+ export -f module
+ export -f ml
+ unset export_module
+ '[' -n '' ']'
+ '[' 4 -ge 3 ']'
+ '[' -r /usr/share/lmod/lmod/init/lmod_bash_completions ']'
+ '[' -n '' ']'
+ ml Singularity/2.5.2-GCC-5.4.0-2.26
++ /usr/share/lmod/lmod/libexec/ml_cmd Singularity/2.5.2-GCC-5.4.0-2.26
+ eval module load ''\''Singularity/2.5.2-GCC-5.4.0-2.26'\'''
++ module load Singularity/2.5.2-GCC-5.4.0-2.26
+++ /usr/share/lmod/lmod/libexec/lmod bash load Singularity/2.5.2-GCC-5.4.0-2.26
++ eval 'LMOD_DEFAULT_MODULEPATH="/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core";' export 'LMOD_DEFAULT_MODULEPATH;' 'M
+++ LMOD_DEFAULT_MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export LMOD_DEFAULT_MODULEPATH
+++ MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export MODULEPATH
+++ CPATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/include:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/include:/apps/software/binutils/2.26-GCCcore-5.4.0/include:/apps/software
+++ export CPATH
+++ EBDEVELBINUTILS=/apps/software/binutils/2.26-GCCcore-5.4.0/easybuild/binutils-2.26-GCCcore-5.4.0-easybuild-devel
+++ export EBDEVELBINUTILS
+++ EBDEVELGCC=/apps/software/GCC/5.4.0-2.26/easybuild/GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELGCC
+++ EBDEVELGCCCORE=/apps/software/GCCcore/5.4.0/easybuild/GCCcore-5.4.0-easybuild-devel
+++ export EBDEVELGCCCORE
+++ EBDEVELLIBARCHIVE=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/easybuild/libarchive-3.3.1-GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELLIBARCHIVE
+++ EBDEVELSINGULARITY=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/easybuild/Singularity-2.5.2-GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELSINGULARITY
+++ EBROOTBINUTILS=/apps/software/binutils/2.26-GCCcore-5.4.0
+++ export EBROOTBINUTILS
+++ EBROOTGCC=/apps/software/GCCcore/5.4.0
+++ export EBROOTGCC
+++ EBROOTGCCCORE=/apps/software/GCCcore/5.4.0
+++ export EBROOTGCCCORE
+++ EBROOTLIBARCHIVE=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26
+++ export EBROOTLIBARCHIVE
+++ EBROOTSINGULARITY=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26
+++ export EBROOTSINGULARITY
+++ EBVERSIONBINUTILS=2.26
+++ export EBVERSIONBINUTILS
+++ EBVERSIONGCC=5.4.0
+++ export EBVERSIONGCC
+++ EBVERSIONGCCCORE=5.4.0
+++ export EBVERSIONGCCCORE
+++ EBVERSIONLIBARCHIVE=3.3.1
+++ export EBVERSIONLIBARCHIVE
+++ EBVERSIONSINGULARITY=2.5.2
+++ export EBVERSIONSINGULARITY
+++ LD_LIBRARY_PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/lib:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib:/apps/software/binutils/2.26-GCCcore-5.4.0/lib:/apps/software/G
+++ export LD_LIBRARY_PATH
+++ LIBRARY_PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/lib:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib:/apps/software/binutils/2.26-GCCcore-5.4.0/lib:/apps/software/GCCc
+++ export LIBRARY_PATH
+++ LMOD_DEFAULT_MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export LMOD_DEFAULT_MODULEPATH
+++ LOADEDMODULES=GCCcore/5.4.0:binutils/2.26-GCCcore-5.4.0:GCC/5.4.0-2.26:libarchive/3.3.1-GCC-5.4.0-2.26:Singularity/2.5.2-GCC-5.4.0-2.26
+++ export LOADEDMODULES
+++ MANPATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/share/man:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/share/man:/apps/software/binutils/2.26-GCCcore-5.4.0/share/man:/apps/
+++ export MANPATH
+++ MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export MODULEPATH
+++ PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binutils/2.26-GCCcore-5.4.0/bin:/apps/software/GCCcore/5.4.
+++ export PATH
+++ PKG_CONFIG_PATH=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib/pkgconfig
+++ export PKG_CONFIG_PATH
+++ _LMFILES_=/apps/modules/all/GCCcore/5.4.0:/apps/modules/all/binutils/2.26-GCCcore-5.4.0:/apps/modules/all/GCC/5.4.0-2.26:/apps/modules/all/libarchive/3.3.1-GCC-5.4.0-2.26.lua:/apps/m
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJhY3RpdmVTaXplIl09NSxiYXNlTXBhdGhBPXsiL2FwcHMvbW9kdWxlcy9hbGwiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4IiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlI
+++ export _ModuleTable001_
+++ _ModuleTable002_=Y3RpdmUiLH0sR0NDY29yZT17WyJGTiJdPSIvYXBwcy9tb2R1bGVzL2FsbC9HQ0Njb3JlLzUuNC4wIixbImRlZmF1bHQiXT0wLFsiZnVsbE5hbWUiXT0iR0NDY29yZS81LjQuMCIsWyJsb2FkT3JkZXIiXT0xLHByb3BUP
+++ export _ModuleTable002_
+++ _ModuleTable003_=aWxzPXtbIkZOIl09Ii9hcHBzL21vZHVsZXMvYWxsL2JpbnV0aWxzLzIuMjYtR0NDY29yZS01LjQuMCIsWyJkZWZhdWx0Il09MCxbImZ1bGxOYW1lIl09ImJpbnV0aWxzLzIuMjYtR0NDY29yZS01LjQuMCIsWyJsb2FkT
+++ export _ModuleTable003_
+++ _ModuleTable004_=aXZlIix9LH0sbXBhdGhBPXsiL2FwcHMvbW9kdWxlcy9hbGwiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4IiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlIiwiL3Vzci9zaGFyZS9sbW9kL2xtb2QvbW9kd
+++ export _ModuleTable004_
+++ _ModuleTable_Sz_=4
+++ export _ModuleTable_Sz_
++ '[' 0 = 0 ']'
+++ : -s sh
++ eval
+ export PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binu
+ PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binutils/2.
+ singularity exec /fast/users/a1222423/eager/work/singularity/nf-core-eager.img /bin/bash -c 'pwd; cd /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f; pwd; ls -al; /bi
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
total 52
drwxrwxr-x 2 a1222423 a1222423  4096 Nov  7 05:09 .
drwxrwxr-x 3 a1222423 a1222423  4096 Nov  5 13:40 ..
-rw-rw-r-- 1 a1222423 a1222423     0 Nov  5 13:40 .command.begin
-rw-rw-r-- 1 a1222423 a1222423   234 Nov  5 13:40 .command.err
-rw-rw-r-- 1 a1222423 a1222423  1353 Nov  5 13:40 .command.log
-rw-rw-r-- 1 a1222423 a1222423     0 Nov  5 13:40 .command.out
-rw-rw-r-- 1 a1222423 a1222423  2591 Nov  7 05:06 .command.run
-rw-rw-r-- 1 a1222423 a1222423   787 Nov  5 13:40 .command.sh
-rw-rw-r-- 1 a1222423 a1222423  3495 Nov  5 13:40 .command.stub
-rw-rw-r-- 1 a1222423 a1222423     3 Nov  5 13:40 .exitcode
-rw-rw-r-- 1 a1222423 a1222423 15383 Nov  7 05:09 slurm-12963871.out
-rw-rw-r-- 1 a1222423 a1222423   414 Nov  7 05:08 test-bowen.sh

The output above showed the binding worked well.

However, nextflow still have the binding issue

[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ echo $SINGULARITY_BINDPATH                                                                                                                 /fast/users/a1222423/eager/work,/data
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true  -profile phoenix -r dev -name $(date +%Y%m%d%H%M)
N E X T F L O W  ~  version 18.10.1
Launching `nf-core/eager` [201811081350] - revision: a827830491 [dev]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [a2b0c7c40a]
=========================================
 nf-core/eager v2.1.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name       : 201811081350
Reads          : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work
Container Engine: singularity
Container      : shub://nf-core/eager
Current home   : /home/a1222423
Current user   : a1222423
Current path   : /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
Script dir     : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : phoenix
=========================================
[warm up] executor > slurm
[d4/ee3b47] Submitted process > get_software_versions
[37/d40954] Submitted process > fastqc (NGSD2_r2)
[24/a5fd8c] Submitted process > adapter_removal (NGSD2_r2)
[f8/caaf83] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Process `get_software_versions` terminated with an error exit status (127)

Command executed:

  echo 2.1.0dev &> v_pipeline.txt
  echo 18.10.1 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  bwa &> v_bwa.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  AdapterRemoval -version  &> v_adapterremoval.txt 2>&1 || true
  picard MarkDuplicates --version &> v_markduplicates.txt  2>&1 || true
  dedup -v &> v_dedup.txt 2>&1 || true
  preseq &> v_preseq.txt 2>&1 || true
  gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
  vcf2genome &> v_vcf2genome.txt 2>&1 || true  
  fastp --version &> v_fastp.txt 2>&1 || true  
  bam --version &> v_bamutil.txt 2>&1 || true  
  qualimap --version &> v_qualimap.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work/d4/ee3b4742983a9d9387e27bb9b49312: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work/d4/ee3b4742983a9d9387e27bb9b49312

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[37/d40954] NOTE: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete

I also ran another singularity image for a Computational Fluid Dynamics (CFD) application named OpenFOAM under /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f with @yassineS account.

The working directory is /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/pitzDaily and both input and output are under it.

It worked well.

All above indicates the issue is on nextflow side rather than singularity configuration, possibly the part integration with singularity or nextflow configuration (nextflow might redefine or not propagate $SINGULARITY_BINDPATH (?)), unless some bugs inside singularity itself

Cheers,

Bowen

Edited history:

Edited format and typos

apeltzer commented 5 years ago

Aah, I see!

I added the envWhitelist=SINGULARITY_BINDPATH option in the phoenix profile for this pipeline. @yassineS Could you test this using the -r dev -profile phoenix branch?

That could resolve this issue then, just make sure you have set your SINGULARITY_BINDPATH before running the pipeline then.

uoabowen commented 5 years ago

Hi @apeltzer , I bring bad and good news

I tested it with the new profile, it still had the same issue. So I set bind path in singularity.conf to let it binds paths automatically, seems everything works fine.

I searched nextflow's document, and found below under images section on https://www.nextflow.io/docs/latest/singularity.html#images

If your Singularity installation support the user bind control feature, enable the Nextflow support for that by defining the singularity.autoMounts = true setting in the Nextflow configuration file.

Seems add singularity.autoMounts = true in profile or nextflow.config (not sure adding to which is correct or better) may resolve the issue when using the user bind control feature with singularity

apeltzer commented 5 years ago

Hi @uoabowen !

I added the required configuration to the acad-phoenix profile, so using the -profile phoenix should do the trick now!

yassineS commented 5 years ago

That's great!