search

nf-core / ampliseq

Amplicon sequencing analysis workflow using DADA2 and QIIME2
https://nf-co.re/ampliseq
MIT License
187 stars 117 forks source link

Test issue #226

Closed dinhe878 closed 3 years ago

dinhe878 commented 3 years ago

Hi,

I was trying to run test on the pipeline as instructed:

nextflow run nf-core/ampliseq -profile test,singularity

But got error messages at the fastqc step:

Error executing process > 'fastqc (1)'

Caused by: Process fastqc (1) terminated with an error exit status (2)

Command executed:

fastqc -q 1_S103_L001_R1_001.fastq.gz 1_S103_L001_R2_001.fastq.gz

Command exit status: 2

Command output: (empty)

Command error: INFO: Converting SIF file to temporary sandbox... Can't locate warnings.pm in @INC (you may need to install the warnings module) (@INC contains: /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/site_perl/5.26.2/x86_64-linux-thread-multi /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/site_perl/5.26.2 /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/5.26.2/x86_64-linux-thread-multi /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/5.26.2 .) at /opt/conda/envs/nf-core-ampliseq-1.2.0/bin/fastqc line 2. BEGIN failed--compilation aborted at /opt/conda/envs/nf-core-ampliseq-1.2.0/bin/fastqc line 2. INFO: Cleaning up image...

I'm not sure why it complains the missing "warning module", as I run with singularity and shouldn't it contain all the required packages?

Thanks a lot for the help!

emnilsson commented 3 years ago

Hi there! Yes, the singularity image should contain everything you need, but to me it looks like it might be singularity that is causing the issue. Maybe you could try pulling the singularity image beforehand with singularity pull --name nf-core/ampliseq docker://nfcore/ampliseq?

dinhe878 commented 3 years ago

Thanks @emnilsson. So I pulled the image and executed: nextflow run nf-core/ampliseq -profile test -with-singularity $SINGULARITYPATH/ampliseq and got:

Error executing process > 'get_software_versions'

Caused by: Process get_software_versions terminated with an error exit status (127)

Command executed:

echo 1.2.0 > v_pipeline.txt echo 20.10.0 > v_nextflow.txt fastqc --version > v_fastqc.txt multiqc --version > v_multiqc.txt cutadapt --version > v_cutadapt.txt qiime --version > v_qiime.txt scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status: 127

Command output: (empty)

Command error: INFO: Converting SIF file to temporary sandbox... /bin/bash: line 0: cd: /home/projects/ku_00039/people/dinghe/working_dr/amplicon_seq/test/work/ad/f3a8c99bb2ed86c85bc64039300130: No such file or directory /bin/bash: .command.run: No such file or directory INFO: Cleaning up image...

It complains about no such file or directory but it does exist.

Any suggestion? Many thanks!

d4straub commented 3 years ago

There seems to be still no container available. Please execute these commands:

#pull the image:
singularity pull --name nfcore-ampliseq-1.2.0.img docker://nfcore/ampliseq:1.2.0

#just in case the folder structure isn't there yet:
mkdir work
mkdir work/singularity

#copy image to default location:
mv nfcore-ampliseq-1.2.0.img ./work/singularity/

#execute pipeline:
nextflow run nf-core/ampliseq -r 1.2.0 -profile test,singularity
dinhe878 commented 3 years ago

Thanks @d4straub. From your suggestion, however, I got back the original errors:

Error executing process > 'fastqc (1)'

Caused by: Process fastqc (1) terminated with an error exit status (2)

Command executed:

fastqc -q 1_S103_L001_R1_001.fastq.gz 1_S103_L001_R2_001.fastq.gz

Command exit status: 2

Command output: (empty)

Command error: INFO: Converting SIF file to temporary sandbox... Can't locate warnings.pm in @INC (you may need to install the warnings module) (@INC contains: /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/site_perl/5.26.2/x86_64-linux-thread-multi /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/site_perl/5.26.2 /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/5.26.2/x86_64-linux-thread-multi /opt/conda/envs/nf-core-ampliseq-1.2.0/lib/5.26.2 .) at /opt/conda/envs/nf-core-ampliseq-1.2.0/bin/fastqc line 2. BEGIN failed--compilation aborted at /opt/conda/envs/nf-core-ampliseq-1.2.0/bin/fastqc line 2. INFO: Cleaning up image...

d4straub commented 3 years ago

Hi,

this might be a singularity problem. To exclude this, could you (1) test whether you can use the container you downloaded before (e.g. singularity exec work/singularity/nfcore-ampliseq-1.2.0.img qiime) (2) make sure to run the newest version of singularity, (3) use conda with nextflow run nf-core/ampliseq -r 1.2.0 -profile test,conda or docker with nextflow run nf-core/ampliseq -r 1.2.0 -profile test,docker.

dinhe878 commented 3 years ago

Hi @d4straub, thanks for the suggestion. So I did 1) singularity exec work/singularity/nfcore-ampliseq-1.2.0.img qiime and it seems to work although still ended up with some error messages toward the end

INFO: Converting SIF file to temporary sandbox... QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment. Usage: qiime [OPTIONS] COMMAND [ARGS]...

QIIME 2 command-line interface (q2cli)

To get help with QIIME 2, visit https://qiime2.org.

To enable tab completion in Bash, run the following command or add it to your .bashrc/.bash_profile:

  source tab-qiime

To enable tab completion in ZSH, run the following commands or add them to your .zshrc:

  autoload bashcompinit && bashcompinit && source tab-qiime

Options: --version Show the version and exit. --help Show this message and exit.

Commands: info Display information about current deployment. tools Tools for working with QIIME 2 files. dev Utilities for developers and advanced users. alignment Plugin for generating and manipulating alignments. composition Plugin for compositional data analysis. cutadapt Plugin for removing adapter sequences, primers, and other unwanted sequence from sequence data. dada2 Plugin for sequence quality control with DADA2. deblur Plugin for sequence quality control with Deblur. demux Plugin for demultiplexing & viewing sequence quality. diversity Plugin for exploring community diversity. emperor Plugin for ordination plotting with Emperor. feature-classifier Plugin for taxonomic classification. feature-table Plugin for working with sample by feature tables. fragment-insertion Plugin for extending phylogenies. gneiss Plugin for building compositional models. longitudinal Plugin for paired sample and time series analyses. metadata Plugin for working with Metadata. phylogeny Plugin for generating and manipulating phylogenies. quality-control Plugin for quality control of feature and sequence data. quality-filter Plugin for PHRED-based filtering and trimming. sample-classifier Plugin for machine learning prediction of sample metadata. taxa Plugin for working with feature taxonomy annotations. vsearch Plugin for clustering and dereplicating with vsearch. INFO: Cleaning up image... ERROR: failed to delete container image tempDir /home/people/dinghe/ku_00039/people/dinghe/container/singularity/.singularity/rootfs-023800771: unlinkat /home/people/dinghe/ku_00039/people/dinghe/container/singularity/.singularity/rootfs-023800771/root: directory not empty

2) I used singularity/3.7.1 which I believe to be the latest version? By the way I used nextflow/20.10.0 not nextflow/21.01.0-edge.

3) The HPC I am testing it on does not allow docker. I tested with conda and it worked fine.

So I guess it's singularity problem? I still prefer to use singularity if I can. Thanks a lot for the help!

d4straub commented 3 years ago

  1. I just tested also singularity exec /nfsmounts/container/nfcore-ampliseq-1.2.0.img qiime on our cluster just to make sure, and there was no error. The last lines

    INFO: Cleaning up image...
ERROR: failed to delete container image tempDir /home/people/dinghe/ku_00039/people/dinghe/container/singularity/.singularity/rootfs-023800771: unlinkat /home/people/dinghe/ku_00039/people/dinghe/container/singularity/.singularity/rootfs-023800771/root: directory not empty

    did not appear for me. So this seems indeed a problem with setting up singularity or write rights (ERROR: failed to delete container image). Your system administrator might be able to solve this.

  2. 3.7.1 should be fine, we use currently 3.4.2 on our cluster, this works as well. Just wanted to make sure you are not using an ancient version. nextflow 20.10.0 is fine, you should not need any edge version.

  3. How about the container software podman? Personally did not use it, so I cannot recommend it, but it is supposed to work as well with -profile podman (and if installed, of course). Otherwise conda is ok as last resort.

dinhe878 commented 3 years ago

Thanks @d4straub hopefully I can find a better solution down the line.

d4straub commented 3 years ago

Feel free to open another issue in case you have question, I am closing this one.