Closed apeltzer closed 5 years ago
Can you confirm that singularity is installed properly?
Command exit status:
127
Command output:
(empty)
Command error:
env: singularity: No such file or directory
This suggests its not installed on your cluster and Nextflow can't call singularity properly. Maybe some missing module activation? You can by the way also simply run like this:
-profile singularity
without requiring the -with-Singularity
option at all. The image will then be pulled automatically and stored in the work
directory of the current run.
I have rerun with the changes suggested above and on the other thread, I still get the 127 error:
N E X T F L O W ~ version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: 4ac6d7a330 [dev]
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
=========================================
nf-core/eager v2.0.0dev
=========================================
Pipeline Name : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name : textEager3
Reads : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref : false
Data Type : Paired-End
Max Memory : 32G
Max CPUs : 4
Max Time : 12.h
Output dir : ./results
Working dir : /fast/users/a1222423/eager/work
Container Engine: singularity
Container : shub://nf-core/eager:latest
Current home : /home/a1222423
Current user : a1222423
Current path : /home/a1222423/fastdir/eager
Script dir : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : singularity
=========================================
[warm up] executor > SLURM
[da/19ef48] Submitted process > fastqc (NGSD2_r2)
[4b/c2f569] Submitted process > get_software_versions
[38/a1632e] Submitted process > adapter_removal (NGSD2_r2)
[10/71f24c] Submitted process > output_documentation
ERROR ~ Error executing process > 'fastqc (NGSD2_r2)'
Caused by:
Process `fastqc (NGSD2_r2)` terminated with an error exit status (127)
Command executed:
fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz
Command exit status:
127
Command output:
(empty)
Command error:
/bin/bash: line 0: cd: /fast/users/a1222423/eager/work/da/19ef48c42b7eda36c089592a5536f6: No such file or directory
/bin/bash: .command.stub: No such file or directory
Work dir:
/fast/users/a1222423/eager/work/da/19ef48c42b7eda36c089592a5536f6
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
-- Check '.nextflow.log' file for details
[nf-core/eager] Pipeline Complete
Is singularity present on that machine?
Yes!
$ singularity --version
2.5.2-dist
Okay, it looks a bit weird because the error here:
Process `fastqc (NGSD2_r2)` terminated with an error exit status (127)
Command executed:
fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz
Command exit status:
127
Suggests via exit code = 127
that fastqc
can't be found.
Also, this here:
Config Profile : singularity
=========================================
[warm up] executor > SLURM
states that you use a SLURM scheduler but didn't specify a profile for this. Do you use your own local configuration to submit jobs to a scheduler? In case so, I can add a configuration for your local hpc environment that should load required modules e.g. singularity automatically, as it could be the case it doesn't load singularity on cluster nodes right now.
There’s no special configuration required to launch a slurm job on our HPC. Does NF require any special configuration?
Y
Cheers, Yassine Souilmi
From: Alexander Peltzer notifications@github.com Sent: Friday, November 2, 2018 11:53:19 PM To: nf-core/eager Cc: Yassine Souilmi; Mention Subject: Re: [nf-core/eager] Singularity missing? (#68)
Okay, it looks a bit weird because the error here:
Process fastqc (NGSD2_r2)
terminated with an error exit status (127)
Command executed:
fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz
Command exit status: 127
Suggests via exit code = 127 that fastqc can't be found.
Also, this here:
[warm up] executor > SLURM
states that you use a SLURM scheduler but didn't specify a profile for this. Do you use your own local configuration to submit jobs to a scheduler? In case so, I can add a configuration for your local hpc environment that should load required modules e.g. singularity automatically, as it could be the case it doesn't load singularity on cluster nodes right now.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/nf-core/eager/issues/68#issuecomment-435378698, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABr4IVBd7-pBmQR2c2S2Ptr9aIBQ3HL6ks5urEdHgaJpZM4YDlNE.
Depends on the cluster quite a bit! Your error message could be explained by submitting jobs to slurm / cluster nodes, but without loading a Singularity module there and thus the tools inside the container cannot be resolved/used (error 127).
How do you ensure that Singularity is up and running on the cluster? Do you load a module or something similar?
Yes! I load the module and double check that it’s callable. Never faced an issue where a module needs to be loaded on the compute nodes.
I have been doing some digging, and the problem might be similar to (https://github.com/nf-core/rnaseq/issues/29#issuecomment-398706617 https://github.com/nf-core/rnaseq/issues/29#issuecomment-398706617). But I am not really sure.
Thanks for your help.
On 4 Nov 2018, at 8:27 pm, Alexander Peltzer <notifications@github.com mailto:notifications@github.com> wrote:
Depends on the cluster quite a bit! Your error message could be explained by submitting jobs to slurm / cluster nodes, but without loading a Singularity module there and thus the tools inside the container cannot be resolved/used (error 127).
How do you ensure that Singularity is up and running on the cluster? Do you load a module or something similar?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nf-core/eager/issues/68#issuecomment-435656270, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr4IVthTf9vtO6PrNkNmGvO8eWjJqCLks5urrnxgaJpZM4YDlNE.
Hey!
Can you let me know how you load the module and maybe just give me a name for the cluster? I can then come up with a -profile <your_cluster_name>
that should run :+1:
module load modules/singularity/2.5.2
The cluster is called Phoenix, and I load the module using the following command:
module load Singularity/2.5.2-GCC-5.4.0-2.26
On 4 Nov 2018, at 8:33 pm, Alexander Peltzer <notifications@github.com mailto:notifications@github.com> wrote:
Hey! Can you let me know how you load the module and maybe just give me a name for the cluster? I can then come up with a -profile
that should run 👍 module load modules/singularity/2.5.2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nf-core/eager/issues/68#issuecomment-435656666, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr4ITNGP7cNkVm8gX1dw-PxLr51TWF8ks5urruBgaJpZM4YDlNE.
Thanks a bunch!
I added a config profile for you to test - its called -profile phoenix
and is currently only found in the dev branch, therefore run your command using this:
nextflow run [your previous options] -profile phoenix -r dev
Unfortunately, this didn't work. It first returned the following error:
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /fast/users/a1222423/eager/work/singularity
Pulling Singularity image docker://nfcore/eager:latest [cache /fast/users/a1222423/eager/work/singularity/nfcore-eager-latest.img]
ERROR ~ Error executing process > 'get_software_versions'
Caused by:
Failed to pull singularity image
command: singularity pull --name nfcore-eager-latest.img docker://nfcore/eager:latest > /dev/null
status : 255
message:
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
ERROR: You must install squashfs-tools to build images
ABORT: Aborting with RETVAL=255
ERROR: pulling container failed!
-- Check '.nextflow.log' file for details
[nf-core/eager] Pipeline Complete
Which I tried to circumvent by downloading the image manually, and than rerunning the workflow:
$ singularity pull --name nfcore-eager.img shub://nf-core/eager:latest
Progress |===================================| 100.0%
Done. Container is at: /fast/users/a1222423/eager/nfcore-eager.img
$
$
$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true -profile phoenix -with-singularity nfcore-eager.img -name textEager3
N E X T F L O W ~ version 0.32.0
Project `nf-core/eager` currently is sticked on revision: dev -- you need to specify explicitly a revision with the option `-r` to use it
[a1222423@l03 eager]$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true -profile phoenix -with-singularity nfcore-eager.img -r dev -name textEager3
N E X T F L O W ~ version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: a6bc45406e [dev]
=========================================
nf-core/eager v2.1.0dev
=========================================
Pipeline Name : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name : textEager3
Reads : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref : false
Data Type : Paired-End
Max Memory : 32G
Max CPUs : 4
Max Time : 12.h
Output dir : ./results
Working dir : /fast/users/a1222423/eager/work
Container Engine: singularity
Container : nfcore-eager.img
Current home : /home/a1222423
Current user : a1222423
Current path : /home/a1222423/fastdir/eager
Script dir : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : phoenix
=========================================
[warm up] executor > slurm
[2b/b9706a] Submitted process > get_software_versions
[8e/3ae677] Submitted process > fastqc (NGSD2_r2)
[68/3dd262] Submitted process > adapter_removal (NGSD2_r2)
[c1/03664f] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'
Caused by:
Process `get_software_versions` terminated with an error exit status (127)
Command executed:
echo 2.1.0dev &> v_pipeline.txt
echo 0.32.0 &> v_nextflow.txt
fastqc --version &> v_fastqc.txt 2>&1 || true
multiqc --version &> v_multiqc.txt 2>&1 || true
bwa &> v_bwa.txt 2>&1 || true
samtools --version &> v_samtools.txt 2>&1 || true
AdapterRemoval -version &> v_adapterremoval.txt 2>&1 || true
picard MarkDuplicates --version &> v_markduplicates.txt 2>&1 || true
dedup -v &> v_dedup.txt 2>&1 || true
preseq &> v_preseq.txt 2>&1 || true
gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
vcf2genome &> v_vcf2genome.txt 2>&1 || true
fastp --version &> v_fastp.txt 2>&1 || true
bam --version &> v_bamutil.txt 2>&1 || true
qualimap --version &> v_qualimap.txt 2>&1 || true
scrape_software_versions.py &> software_versions_mqc.yaml
Command exit status:
127
Command output:
(empty)
Command error:
/bin/bash: line 0: cd: /fast/users/a1222423/eager/work/2b/b9706a04822a46b640fa5fc67d10c6: No such file or directory
/bin/bash: .command.stub: No such file or directory
Work dir:
/fast/users/a1222423/eager/work/2b/b9706a04822a46b640fa5fc67d10c6
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
-- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[8e/3ae677] NOTE: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete
Ok, the first issue can be resolved by installing squashfs-tools
on the server once. Maybe ask IT to do so, we had to as well ;-)
The second issue seems indeed to be related to the link you posted: https://github.com/nf-core/rnaseq/issues/29#issuecomment-398706617
It looks like your cluster admin (?) didn't specify certain configuration parameters for singularity to access certain paths on your system. For example, we have a system where we can only run images and access files stored under /beegfs/<....>
and this has to be configured in the Singularity installation once for all users.
I can help you in getting that up and communicate it appropriately if you want to, but we'd need to have a shared screen session / Skype to talk about this ...
Sounds good to me, and thank you again for your support!
No worries, happy to have some people testing the pipeline and giving feedback like this - thanks for testing and send me a DM/email if we should do a screen share or similar session to figure out whats going on.
Heads up to document for future cases what goes on:
Head node uses a different home than worker nodes on cluster, this leads to entire work directories not being readable, thus the error of missing things. Had similar issues on another cluster before, where the homes were linked as
/home-link/<yourusername>
and then on the node they were present as /home/<yourusername>
...
Hi @apeltzer,
I am the guy who installed and configured singularity on Phoenix which @yassineS is using.
We use $SINGULARITY_BINDPATH to bind directory, see https://www.sylabs.io/guides/2.6/user-guide/bind_paths_and_mounts.html#binding-with-overlay for details
This is my test script
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ echo $SINGULARITY_BINDPATH
/fast/users/a1222423/eager/work,/data
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ cat test-bowen.sh
#!/bin/bash -x
#SBATCH -p test
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --mem=1G
#SBATCH --time=15:00
ml Singularity/2.5.2-GCC-5.4.0-2.26
export PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:$PATH
singularity exec /fast/users/a1222423/eager/work/singularity/nf-core-eager.img /bin/bash -c "pwd; cd $PWD; pwd; ls -al; /bin/bash /fast/users/a1222423/eager/work/8b//12a213d46743cf1ccdd3e71e23f23f/.command.stub"
This is what I got
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ cat slurm-12963871.out
+ LMOD_PKG=/usr/share/lmod/lmod
+ LMOD_DIR=/usr/share/lmod/lmod/libexec
+ LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod
+ MODULESHOME=/usr/share/lmod/lmod
+ export LMOD_PKG
+ export LMOD_CMD
+ export LMOD_DIR
+ export MODULESHOME
+ LMOD_VERSION=6.3.4
+ export LMOD_VERSION
+ '[' : '!=' : ']'
+ unalias ml
+ true
++ /usr/bin/tr '[:upper:]' '[:lower:]'
++ echo YES
+ export_module=yes
+ '[' -n '4.2.46(2)-release' -a yes '!=' no ']'
+ export -f module
+ export -f ml
+ unset export_module
+ '[' -n '' ']'
+ '[' 4 -ge 3 ']'
+ '[' -r /usr/share/lmod/lmod/init/lmod_bash_completions ']'
+ '[' -n '' ']'
+ ml Singularity/2.5.2-GCC-5.4.0-2.26
++ /usr/share/lmod/lmod/libexec/ml_cmd Singularity/2.5.2-GCC-5.4.0-2.26
+ eval module load ''\''Singularity/2.5.2-GCC-5.4.0-2.26'\'''
++ module load Singularity/2.5.2-GCC-5.4.0-2.26
+++ /usr/share/lmod/lmod/libexec/lmod bash load Singularity/2.5.2-GCC-5.4.0-2.26
++ eval 'LMOD_DEFAULT_MODULEPATH="/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core";' export 'LMOD_DEFAULT_MODULEPATH;' 'M
+++ LMOD_DEFAULT_MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export LMOD_DEFAULT_MODULEPATH
+++ MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export MODULEPATH
+++ CPATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/include:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/include:/apps/software/binutils/2.26-GCCcore-5.4.0/include:/apps/software
+++ export CPATH
+++ EBDEVELBINUTILS=/apps/software/binutils/2.26-GCCcore-5.4.0/easybuild/binutils-2.26-GCCcore-5.4.0-easybuild-devel
+++ export EBDEVELBINUTILS
+++ EBDEVELGCC=/apps/software/GCC/5.4.0-2.26/easybuild/GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELGCC
+++ EBDEVELGCCCORE=/apps/software/GCCcore/5.4.0/easybuild/GCCcore-5.4.0-easybuild-devel
+++ export EBDEVELGCCCORE
+++ EBDEVELLIBARCHIVE=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/easybuild/libarchive-3.3.1-GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELLIBARCHIVE
+++ EBDEVELSINGULARITY=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/easybuild/Singularity-2.5.2-GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELSINGULARITY
+++ EBROOTBINUTILS=/apps/software/binutils/2.26-GCCcore-5.4.0
+++ export EBROOTBINUTILS
+++ EBROOTGCC=/apps/software/GCCcore/5.4.0
+++ export EBROOTGCC
+++ EBROOTGCCCORE=/apps/software/GCCcore/5.4.0
+++ export EBROOTGCCCORE
+++ EBROOTLIBARCHIVE=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26
+++ export EBROOTLIBARCHIVE
+++ EBROOTSINGULARITY=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26
+++ export EBROOTSINGULARITY
+++ EBVERSIONBINUTILS=2.26
+++ export EBVERSIONBINUTILS
+++ EBVERSIONGCC=5.4.0
+++ export EBVERSIONGCC
+++ EBVERSIONGCCCORE=5.4.0
+++ export EBVERSIONGCCCORE
+++ EBVERSIONLIBARCHIVE=3.3.1
+++ export EBVERSIONLIBARCHIVE
+++ EBVERSIONSINGULARITY=2.5.2
+++ export EBVERSIONSINGULARITY
+++ LD_LIBRARY_PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/lib:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib:/apps/software/binutils/2.26-GCCcore-5.4.0/lib:/apps/software/G
+++ export LD_LIBRARY_PATH
+++ LIBRARY_PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/lib:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib:/apps/software/binutils/2.26-GCCcore-5.4.0/lib:/apps/software/GCCc
+++ export LIBRARY_PATH
+++ LMOD_DEFAULT_MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export LMOD_DEFAULT_MODULEPATH
+++ LOADEDMODULES=GCCcore/5.4.0:binutils/2.26-GCCcore-5.4.0:GCC/5.4.0-2.26:libarchive/3.3.1-GCC-5.4.0-2.26:Singularity/2.5.2-GCC-5.4.0-2.26
+++ export LOADEDMODULES
+++ MANPATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/share/man:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/share/man:/apps/software/binutils/2.26-GCCcore-5.4.0/share/man:/apps/
+++ export MANPATH
+++ MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export MODULEPATH
+++ PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binutils/2.26-GCCcore-5.4.0/bin:/apps/software/GCCcore/5.4.
+++ export PATH
+++ PKG_CONFIG_PATH=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib/pkgconfig
+++ export PKG_CONFIG_PATH
+++ _LMFILES_=/apps/modules/all/GCCcore/5.4.0:/apps/modules/all/binutils/2.26-GCCcore-5.4.0:/apps/modules/all/GCC/5.4.0-2.26:/apps/modules/all/libarchive/3.3.1-GCC-5.4.0-2.26.lua:/apps/m
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJhY3RpdmVTaXplIl09NSxiYXNlTXBhdGhBPXsiL2FwcHMvbW9kdWxlcy9hbGwiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4IiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlI
+++ export _ModuleTable001_
+++ _ModuleTable002_=Y3RpdmUiLH0sR0NDY29yZT17WyJGTiJdPSIvYXBwcy9tb2R1bGVzL2FsbC9HQ0Njb3JlLzUuNC4wIixbImRlZmF1bHQiXT0wLFsiZnVsbE5hbWUiXT0iR0NDY29yZS81LjQuMCIsWyJsb2FkT3JkZXIiXT0xLHByb3BUP
+++ export _ModuleTable002_
+++ _ModuleTable003_=aWxzPXtbIkZOIl09Ii9hcHBzL21vZHVsZXMvYWxsL2JpbnV0aWxzLzIuMjYtR0NDY29yZS01LjQuMCIsWyJkZWZhdWx0Il09MCxbImZ1bGxOYW1lIl09ImJpbnV0aWxzLzIuMjYtR0NDY29yZS01LjQuMCIsWyJsb2FkT
+++ export _ModuleTable003_
+++ _ModuleTable004_=aXZlIix9LH0sbXBhdGhBPXsiL2FwcHMvbW9kdWxlcy9hbGwiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4IiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlIiwiL3Vzci9zaGFyZS9sbW9kL2xtb2QvbW9kd
+++ export _ModuleTable004_
+++ _ModuleTable_Sz_=4
+++ export _ModuleTable_Sz_
++ '[' 0 = 0 ']'
+++ : -s sh
++ eval
+ export PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binu
+ PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binutils/2.
+ singularity exec /fast/users/a1222423/eager/work/singularity/nf-core-eager.img /bin/bash -c 'pwd; cd /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f; pwd; ls -al; /bi
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
total 52
drwxrwxr-x 2 a1222423 a1222423 4096 Nov 7 05:09 .
drwxrwxr-x 3 a1222423 a1222423 4096 Nov 5 13:40 ..
-rw-rw-r-- 1 a1222423 a1222423 0 Nov 5 13:40 .command.begin
-rw-rw-r-- 1 a1222423 a1222423 234 Nov 5 13:40 .command.err
-rw-rw-r-- 1 a1222423 a1222423 1353 Nov 5 13:40 .command.log
-rw-rw-r-- 1 a1222423 a1222423 0 Nov 5 13:40 .command.out
-rw-rw-r-- 1 a1222423 a1222423 2591 Nov 7 05:06 .command.run
-rw-rw-r-- 1 a1222423 a1222423 787 Nov 5 13:40 .command.sh
-rw-rw-r-- 1 a1222423 a1222423 3495 Nov 5 13:40 .command.stub
-rw-rw-r-- 1 a1222423 a1222423 3 Nov 5 13:40 .exitcode
-rw-rw-r-- 1 a1222423 a1222423 15383 Nov 7 05:09 slurm-12963871.out
-rw-rw-r-- 1 a1222423 a1222423 414 Nov 7 05:08 test-bowen.sh
The output above showed the binding worked well.
However, nextflow still have the binding issue
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ echo $SINGULARITY_BINDPATH /fast/users/a1222423/eager/work,/data
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true -profile phoenix -r dev -name $(date +%Y%m%d%H%M)
N E X T F L O W ~ version 18.10.1
Launching `nf-core/eager` [201811081350] - revision: a827830491 [dev]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [a2b0c7c40a]
=========================================
nf-core/eager v2.1.0dev
=========================================
Pipeline Name : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name : 201811081350
Reads : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref : false
Data Type : Paired-End
Max Memory : 32G
Max CPUs : 4
Max Time : 12.h
Output dir : ./results
Working dir : /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work
Container Engine: singularity
Container : shub://nf-core/eager
Current home : /home/a1222423
Current user : a1222423
Current path : /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
Script dir : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : phoenix
=========================================
[warm up] executor > slurm
[d4/ee3b47] Submitted process > get_software_versions
[37/d40954] Submitted process > fastqc (NGSD2_r2)
[24/a5fd8c] Submitted process > adapter_removal (NGSD2_r2)
[f8/caaf83] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'
Caused by:
Process `get_software_versions` terminated with an error exit status (127)
Command executed:
echo 2.1.0dev &> v_pipeline.txt
echo 18.10.1 &> v_nextflow.txt
fastqc --version &> v_fastqc.txt 2>&1 || true
multiqc --version &> v_multiqc.txt 2>&1 || true
bwa &> v_bwa.txt 2>&1 || true
samtools --version &> v_samtools.txt 2>&1 || true
AdapterRemoval -version &> v_adapterremoval.txt 2>&1 || true
picard MarkDuplicates --version &> v_markduplicates.txt 2>&1 || true
dedup -v &> v_dedup.txt 2>&1 || true
preseq &> v_preseq.txt 2>&1 || true
gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
vcf2genome &> v_vcf2genome.txt 2>&1 || true
fastp --version &> v_fastp.txt 2>&1 || true
bam --version &> v_bamutil.txt 2>&1 || true
qualimap --version &> v_qualimap.txt 2>&1 || true
scrape_software_versions.py &> software_versions_mqc.yaml
Command exit status:
127
Command output:
(empty)
Command error:
/bin/bash: line 0: cd: /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work/d4/ee3b4742983a9d9387e27bb9b49312: No such file or directory
/bin/bash: .command.stub: No such file or directory
Work dir:
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work/d4/ee3b4742983a9d9387e27bb9b49312
Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`
-- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[37/d40954] NOTE: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete
I also ran another singularity image for a Computational Fluid Dynamics (CFD) application named OpenFOAM under /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f with @yassineS account.
The working directory is /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/pitzDaily and both input and output are under it.
It worked well.
All above indicates the issue is on nextflow side rather than singularity configuration, possibly the part integration with singularity or nextflow configuration (nextflow might redefine or not propagate $SINGULARITY_BINDPATH (?)), unless some bugs inside singularity itself
Cheers,
Bowen
Edited history:
Edited format and typos
Aah, I see!
I added the envWhitelist=SINGULARITY_BINDPATH
option in the phoenix profile for this pipeline.
@yassineS Could you test this using the -r dev -profile phoenix
branch?
That could resolve this issue then, just make sure you have set your SINGULARITY_BINDPATH before running the pipeline then.
Hi @apeltzer , I bring bad and good news
I tested it with the new profile, it still had the same issue. So I set bind path in singularity.conf to let it binds paths automatically, seems everything works fine.
I searched nextflow's document, and found below under images section on https://www.nextflow.io/docs/latest/singularity.html#images
If your Singularity installation support the user bind control feature, enable the Nextflow support for that by defining the singularity.autoMounts = true setting in the Nextflow configuration file.
Seems add singularity.autoMounts = true
in profile or nextflow.config (not sure adding to which is correct or better) may resolve the issue when using the user bind control feature with singularity
Hi @uoabowen !
I added the required configuration to the acad-phoenix profile, so using the -profile phoenix
should do the trick now!
That's great!
Originally posted by @yassineS in https://github.com/nf-core/eager/issues/67#issuecomment-434595084