get_software_versions cannot find singularity

[tobsecret]

Nextflow commandline

nextflow run nf-core/rnaseq -profile test,singularity -c my.config where all my.config does is set singularity.autoMounts to true, bc we originally thought it might be a problem.

Error

I'm helping a co-worker run nf-core/rnaseq on his local machine and am running into this problem:

Caused by:
  Process `get_software_versions` terminated with an error exit status (255)

Command executed:

  echo 1.3 &> v_ngi_rnaseq.txt
  echo 19.04.1 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt
  cutadapt --version &> v_cutadapt.txt
  trim_galore --version &> v_trim_galore.txt
  STAR --version &> v_star.txt
  hisat2 --version &> v_hisat2.txt
  stringtie --version &> v_stringtie.txt
  preseq &> v_preseq.txt
  read_duplication.py --version &> v_rseqc.txt
  echo $(bamCoverage --version 2>&1) > v_deeptools.txt
  featureCounts -v &> v_featurecounts.txt
  picard MarkDuplicates --version &> v_markduplicates.txt  || true
  samtools --version &> v_samtools.txt
  multiqc --version &> v_multiqc.txt
  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  255

Command output:
  (empty)

Command error:
  ERROR  : Failed to resolve path to /home/jordan/miniconda3/envs/rnaseq/var/singularity/mnt/container: No such file or directory
  ABORT  : Retval = 255
  

The odd thing is that singularity pulled and converted the container for this pipeline just fine, so I am not exactly sure why it's an issue here.

Full nextflow.log: nextflow.log

Conda env

conda env: conda_list.txt

My colleague still has an old linux version, 14.04, could that have something to do with it?

[apeltzer]

Singularity via conda is a bit problematic - it requires certain execution bits to be set, thus probably not really working reliably (?). At least your error sounds like this. Can't you install Singularity itself as a regular package on that machine?

[tobsecret]

We can try that again, I'll let you know!

[apeltzer]

So I just tried this and couldn't verify it using my own singularity installtion on the local workstation. However, your hint that it might be an Ubuntu 14.04 issue might be correct - could you shell into the container and try to access the path specified in the error message?

/home/jordan/miniconda3/envs/rnaseq/var/singularity/mnt/container 

e.g.

singularity shell container.simg
cd /home/jordan/miniconda3/envs/rnaseq/var/singularity/mnt/container
ls -l 

if that fails, you might need to add proper bind paths to /etc/singularity.d/singularity.conf or using -B directives when running stuff on that system... typical problem coming up quite often.

[apeltzer]

... which I still need to document somewhere because I'm kind of writing this down every 1-2 weeks by now 🤣

[tobsecret]

The path doesn't exist iirc - like that env has singularity but not under that path. But lemme check again! My colleague is now trying to wipe his machine and do a fresh linux install :sweat_smile:

[PrincessDianasaur]

Hey, my stdout is different than the original post, but I suspect I am also facing the binding issue.

Caused by:
  Process `fastqc (8_151109_AC7UAMANXX_P3128_1612)` terminated with an error exit status (255)

Command executed:

  fastqc -q 8_151109_AC7UAMANXX_P3128_1612_1.fastq.gz 8_151109_AC7UAMANXX_P3128_1612_2.fastq.gz

Command exit status:
  255

Command output:
  (empty)

Command error:
  ERROR  : Session directory does not exist: /myHomeDir/miniconda3/var/singularity/mnt/session
  ABORT  : Retval = 255

Work dir:
  /myWorkDir/nf-core_rnaseq_test/work/a4/e3f583b8cdb9a4ecfbcf41b79a784f

I've tried adding on the -B flag but got,

Unknown option: -B -- Check the available commands and options and syntax with 'help'

So I tried to pull the singularity image locally

(base) diana@gimlet:/myWorkDir/nf-core_rnaseq_test$ singularity pull --name nf-core-rnaseq-1.3.img docker://nf-core/rnaseq:1.3
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/nf-core/rnaseq:1.3
ERROR UNAUTHORIZED: authentication required
ERROR Check existence, naming, and permissions
Cleaning up...
ERROR: pulling container failed!

How exactly should I use the -B flag with the rnaseq pipeline? Or is this a different issue entirely?

---

full nextflow.log file: nextflow.log conda env list files: base_env.txt

[ewels]

Hi @PrincessDianasaur,

Have you seen the nf-core troubleshooting docs on this topic? (here) These kinds of problems are usually due to Singularity installation issues. Testing with singularity shell will help with debugging.

Note that the pull command won't work because it's docker://nfcore, not docker://nf-core (apologies, restrictions on names by dockerhub means we can't have a hyphen 😞).

This should work:

singularity pull --name nf-core-rnaseq-1.3.img docker://nfcore/rnaseq:1.3

You can also do nf-core download rnaseq -s with nf-core tools, which figures these things out for you automatically.

[apeltzer]

Hi @PrincessDianasaur - did this help? I have this issue open for our next pipeline release and would like to see / hear if there is some feedback before closing this or whether we need to address something on our side...

[apeltzer]

Ok, if this is still an issue please (re-) open or discuss in a new issue 👍