search

nf-core / ampliseq

Amplicon sequencing analysis workflow using DADA2 and QIIME2
https://nf-co.re/ampliseq
MIT License
184 stars 115 forks source link

ERROR Cannot find image. Is your capitalization correct? #159

Closed MonicaSteffi closed 4 years ago

MonicaSteffi commented 4 years ago

Dear All,

I have two questions.

  1. Why cant I run nf-core/ampliseq without -r 1.0.0? I did execute nf-core/ampliseq without -r in other system
  2. I have encountered the problems in pulling singularity image in nfcore pipeine.

I installed singularity successfully.

singularity --version
2.5.2-master.b94fa35

But when I run the following command,

nextflow run -r 1.0.0 nf-core/ampliseq     -profile singularity     --reads Silke-fastq/ --FW_primer GTGYCAGCMGCCGCGGTAA  --RV_primer GGACTACNVGGGTWTCTAAT  --metadata test.txt   --trunclenf 200   --trunclenr 180  --classifier  classifier_V3_V4.qza --min_frequency 0

N E X T F L O W  ~  version 20.07.1
Launching `nf-core/ampliseq` [insane_perlman] - revision: f0357d61cf [1.0.0]
WARN: Access to undefined parameter `container` -- Initialise it to a default value eg. `params.container = some_value`
=======================================================
                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~'
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

nf-core/ampliseq v1.0.0"
=======================================================
WARN: Access to undefined parameter `max_memory` -- Initialise it to a default value eg. `params.max_memory = some_value`
WARN: Access to undefined parameter `max_cpus` -- Initialise it to a default value eg. `params.max_cpus = some_value`
WARN: Access to undefined parameter `max_time` -- Initialise it to a default value eg. `params.max_time = some_value`
Pipeline Name  : nf-core/ampliseq
Pipeline Version: 1.0.0
Run Name       : insane_perlman
Reads          : /media/ilias/STORAGE/Sandra/nf-core/Silke-fastq/
Max Memory     : null
Max CPUs       : null
Max Time       : null
Output dir     : ./results
Working dir    : /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work
Container Engine: singularity
Container      : (dynamic resolved)
Current home   : /home/ilias
Current user   : ilias
Current path   : /media/ilias/STORAGE/Sandra/nf-core/singularit_test
Script dir     : /home/ilias/.nextflow/assets/nf-core/ampliseq
Config Profile : singularity
=========================================
[-        ] process > get_software_versions        -
[-        ] process > get_software_versions        -
[-        ] process > get_software_versions        -
[-        ] process > fastqc                       -
[-        ] process > get_software_versions        -
[-        ] process > fastqc                       -
[-        ] process > trimming                     -
[-        ] process > multiqc                      -
[-        ] process > qiime_import                 -
[-        ] process > qiime_demux_visualize        -
[-        ] process > dada_trunc_parameter         -
[-        ] process > dada_single                  -
[-        ] process > classifier                   -
[-        ] process > filter_taxa                  -
[-        ] process > export_filtered_dada_output  -
[-        ] process > report_filter_stats          -
[-        ] process > RelativeAbundanceASV         -
[-        ] process > RelativeAbundanceReducedTaxa -
[-        ] process > barplot                      -
[-        ] process > tree                         -
[-        ] process > alpha_rarefaction            -
[-        ] process > combinetable                 -
[-        ] process > diversity_core               -
[-        ] process > metadata_category_all        -
[-        ] process > metadata_category_pairwise   -
[-        ] process > alpha_diversity              -
[-        ] process > beta_diversity               -
[-        ] process > beta_diversity_ordination    -
[-        ] process > prepare_ancom                -
[-        ] process > ancom_tax                    -
[-        ] process > ancom_asv                    -
[-        ] process > output_documentation         -
Pulling Singularity image shub://nf-core/ampliseq:1.0.0 [cache /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/singularity/nf-core-ampliseq-1.0.0.img]
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a different location
Error executing process > 'trimming (188-WT-Starving-Exp-Adlibitum-sv343-ct10_S188)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name nf-core-ampliseq-1.0.0.img shub://nf-core/ampliseq:1.0.0 > /dev/null
  status : 1
  message:
    ERROR Cannot find image. Is your capitalization correct?

I also tried following but still no use: singularity pull --name nfcore-ampliseq.img docker://nfcore/ampliseq

d4straub commented 4 years ago

First of, I do not recommend version 1.0.0, better use 1.1.0 (which is very similar but at least got multiple sequencing runs support).

Why cant I run nf-core/ampliseq without -r 1.0.0? I did execute nf-core/ampliseq without -r in other system

-r is recommended, so please use it. However, it should work without (provided you have done nextflow pull nf-core/ampliseq beforehand).

I have encountered the problems in pulling singularity image in nfcore pipeline.

I just tested singularity pull --name nfcore-ampliseq.img docker://nfcore/ampliseq and it works for me but singularity pull --name nf-core-ampliseq-1.0.0.img shub://nf-core/ampliseq:1.0.0 doesn't work (and actually shouldn't).

However, when I run nextflow run -r 1.0.0 nf-core/ampliseq -profile test,singularity it works flawlessly (pulls image from docker://nfcore/ampliseq:1.0.0). --> I propose either using a newer pipeline version and/or download the image using singularity pull --name nfcore-ampliseq-1.0.0.img docker://nfcore/ampliseq:1.0.0 and then run the pipeline by adding -with-singularity nfcore-ampliseq-1.0.0.img.

edit: of yourse the pipeline and container version have to match!

By the way, I use also like you an outdated singularity version (current version is 3.6):

singularity --version
2.6.1-dist
MonicaSteffi commented 4 years ago

Dear @d4straub

Thank you. Ive updated singularity version and tried as you suggested. But this time I got different error.

nextflow run -r 1.1.0 nf-core/ampliseq -profile test,singularity

Jul-29 11:47:26.344 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 20.07.1 Jul-29 11:47:26.860 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/ilias/.nextflow/assets/nf-core/ampliseq/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/ampliseq.git Jul-29 11:47:26.873 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/ilias/.nextflow/assets/nf-core/ampliseq/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/ampliseq.git Jul-29 11:47:27.119 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/ilias/.nextflow/assets/nf-core/ampliseq/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/ampliseq.git Jul-29 11:47:27.120 [main] INFO nextflow.cli.CmdRun - Launching nf-core/ampliseq [nostalgic_liskov] - revision: f0357d61cf [1.0.0] Jul-29 11:47:27.785 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/ilias/.nextflow/assets/nf-core/ampliseq/nextflow.config Jul-29 11:47:27.786 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/ilias/.nextflow/assets/nf-core/ampliseq/nextflow.config Jul-29 11:47:27.792 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: test,singularity Jul-29 11:47:28.002 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, cfc, uzh, debug, test, conda, singularity, awsbatch, none, docker, binac] Jul-29 11:47:28.069 [main] DEBUG nextflow.Session - Session uuid: 1bb23eec-7ee1-4d86-a3d6-49e1710438ea Jul-29 11:47:28.069 [main] DEBUG nextflow.Session - Run name: nostalgic_liskov Jul-29 11:47:28.070 [main] DEBUG nextflow.Session - Executor pool size: 4 Jul-29 11:47:28.103 [main] DEBUG nextflow.cli.CmdRun - Version: 20.07.1 build 5412 Created: 24-07-2020 15:18 UTC (17:18 CEST) System: Linux 4.15.0-107-generic Runtime: Groovy 2.5.11 on OpenJDK 64-Bit Server VM 11.0.1+13-LTS Encoding: UTF-8 (UTF-8) Process: 26151@ilias-Precision-T1700 [127.0.1.1] CPUs: 4 - Mem: 31.3 GB (1.7 GB) - Swap: 31.9 GB (31.9 GB) Jul-29 11:47:28.116 [main] DEBUG nextflow.Session - Work-dir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work [ext2/ext3] Jul-29 11:47:28.154 [main] DEBUG nextflow.Session - Observer factory: TowerFactory Jul-29 11:47:28.156 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory Jul-29 11:47:28.409 [main] DEBUG nextflow.Session - Session start invoked Jul-29 11:47:28.413 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/results/pipeline_info/nf-core/ampliseq_trace.txt Jul-29 11:47:28.971 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution Jul-29 11:47:28.978 [main] DEBUG nextflow.Session - Workflow process names [dsl1]: RelativeAbundanceReducedTaxa, metadata_category_pairwise, qiime_importdemux_visualize, diversity_core, output_documentation, beta_diversity, prepare_ancom, dada_trunc_parameter, filter_taxa, beta_diversity_ordination, make_SILVA_132_16S_classifier, barplot, classifier, trimming, qiime_import, get_software_versions, multiqc, qiime_demux_visualize, alpha_diversity, metadata_category_all, tree, combinetable, report_filter_stats, alpha_rarefaction, ancom_tax, dada_single, RelativeAbundanceASV, fastqc, ancom_asv, export_filtered_dada_output Jul-29 11:47:29.125 [main] INFO nextflow.Nextflow - ======================================================= ,--./,-. _ _ /,-..--~' |\ | | / ` / \ |) | } { | | | _, __/ | \ |__ `-.,--, .,.,'

nf-core/ampliseq v1.0.0"

Jul-29 11:47:29.138 [main] INFO nextflow.Nextflow - Pipeline Name : nf-core/ampliseq Pipeline Version: 1.0.0 Run Name : nostalgic_liskov Reads : data/*_L001_R{1,2}_001.fastq.gz Max Memory : 6 GB Max CPUs : 2 Max Time : 2d Output dir : ./results Working dir : /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work Container Engine: singularity Container : nfcore/ampliseq:1.0.0 Current home : /home/ilias Current user : ilias Current path : /media/ilias/STORAGE/Sandra/nf-core/singularit_test Script dir : /home/ilias/.nextflow/assets/nf-core/ampliseq Config Profile : test,singularity Jul-29 11:47:29.141 [main] INFO nextflow.Nextflow - ========================================= Jul-29 11:47:29.185 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.185 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.192 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local Jul-29 11:47:29.199 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=4; memory=31.3 GB; capacity=4; pollInterval=100ms; dumpInterval=5m Jul-29 11:47:29.358 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:fastqc matches process fastqc Jul-29 11:47:29.359 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.359 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.366 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.366 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.417 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:multiqc matches process multiqc Jul-29 11:47:29.418 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.418 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.430 [Actor Thread 14] WARN nextflow.container.SingularityCache - Singularity cache directory has not been defined -- Remote image will be stored in the path: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a different location Jul-29 11:47:29.449 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.475 [Actor Thread 14] INFO nextflow.container.SingularityCache - Pulling Singularity image docker://nfcore/ampliseq:1.0.0 [cache /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/singularity/nfcore-ampliseq-1.0.0.img] Jul-29 11:47:29.478 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.557 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.557 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.566 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.566 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.580 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:dada_single matches process dada_single Jul-29 11:47:29.580 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.580 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.601 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:classifier matches process classifier Jul-29 11:47:29.602 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.602 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.608 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.608 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.617 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.617 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.622 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.622 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.626 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.626 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.630 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.630 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.634 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.635 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.645 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:tree matches process tree Jul-29 11:47:29.646 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.646 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.654 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.654 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.661 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.661 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.668 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.669 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.678 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.678 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.681 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.681 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.759 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.759 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.768 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.772 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.783 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.783 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.818 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.818 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.836 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:ancom_tax matches process ancom_tax Jul-29 11:47:29.837 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.837 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.843 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:ancom_asv matches process ancom_asv Jul-29 11:47:29.843 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.844 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.851 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null Jul-29 11:47:29.851 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local' Jul-29 11:47:29.865 [main] DEBUG nextflow.script.ScriptRunner - > Await termination Jul-29 11:47:29.865 [main] DEBUG nextflow.Session - Session await Jul-29 11:47:30.290 [Actor Thread 18] DEBUG n.util.BlockingThreadExecutorFactory - Thread pool name=FileTransfer; maxThreads=12; maxQueueSize=36; keepAlive=1m Jul-29 11:47:30.299 [FileTransfer-thread-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/Metadata.tsv at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/62/335072752510aa562cfe6c055b36f9/Metadata.tsv Jul-29 11:47:30.469 [FileTransfer-thread-2] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/1a_S103_L001_R1_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/9f/b7d41117d3d51e055c144965eb6f01/1a_S103_L001_R1_001.fastq.gz Jul-29 11:47:30.572 [FileTransfer-thread-3] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/1a_S103_L001_R2_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/18/0a0a6fb5c04582bb99134a90ef16d9/1a_S103_L001_R2_001.fastq.gz Jul-29 11:47:30.671 [FileTransfer-thread-4] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/1_S103_L001_R1_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/51/e6e32aed2d552e400b5010b3b36e8a/1_S103_L001_R1_001.fastq.gz Jul-29 11:47:30.771 [FileTransfer-thread-5] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/1_S103_L001_R2_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/2f/5b9368c0eacfbb47dd7fa4dfae787a/1_S103_L001_R2_001.fastq.gz Jul-29 11:47:30.867 [FileTransfer-thread-6] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/2_S115_L001_R1_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/13/1e2e1d7fb408b7ebcd50761f42b836/2_S115_L001_R1_001.fastq.gz Jul-29 11:47:30.963 [FileTransfer-thread-7] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/2_S115_L001_R2_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/cb/42bfb9d046d589c5afc03aee094baf/2_S115_L001_R2_001.fastq.gz Jul-29 11:47:31.063 [FileTransfer-thread-8] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/2_S115_L001_R1_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/13/1e2e1d7fb408b7ebcd50761f42b836/2_S115_L001_R1_001.fastq.gz Jul-29 11:47:31.162 [FileTransfer-thread-9] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/2_S115_L001_R2_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/cb/42bfb9d046d589c5afc03aee094baf/2_S115_L001_R2_001.fastq.gz Jul-29 11:47:31.264 [FileTransfer-thread-10] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/1_S103_L001_R1_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/51/e6e32aed2d552e400b5010b3b36e8a/1_S103_L001_R1_001.fastq.gz Jul-29 11:47:31.373 [FileTransfer-thread-11] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/1_S103_L001_R2_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/2f/5b9368c0eacfbb47dd7fa4dfae787a/1_S103_L001_R2_001.fastq.gz Jul-29 11:47:31.482 [FileTransfer-thread-12] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/2a_S115_L001_R1_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/79/8aeb8c7975749f6d4b51f2ae461ab7/2a_S115_L001_R1_001.fastq.gz Jul-29 11:47:31.576 [FileTransfer-thread-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://github.com/nf-core/test-datasets/raw/ampliseq/testdata/2a_S115_L001_R2_001.fastq.gz at /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/stage/dc/f4d29c76a79a61d2fba6421636ffbe/2a_S115_L001_R2_001.fastq.gz Jul-29 11:52:29.416 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- The following nodes are still active: [process] get_software_versions status=ACTIVE port 0: (cntrl) - ; channel: $

[process] fastqc status=ACTIVE port 0: (queue) closed; channel: - port 1: (cntrl) - ; channel: $

[process] trimming status=ACTIVE port 0: (queue) closed; channel: - port 1: (cntrl) - ; channel: $

[process] multiqc status=ACTIVE port 0: (value) OPEN ; channel: fastqc/ port 1: (value) OPEN ; channel: cutadapt/ port 2: (cntrl) - ; channel: $

[process] qiime_import status=ACTIVE port 0: (value) OPEN ; channel: trimmed port 1: (queue) OPEN ; channel: MATPLOTLIBRC port 2: (cntrl) - ; channel: $

[process] qiime_demux_visualize status=ACTIVE port 0: (queue) OPEN ; channel: demux port 1: (queue) OPEN ; channel: MATPLOTLIBRC port 2: (cntrl) - ; channel: $

[process] dada_trunc_parameter status=ACTIVE port 0: (queue) OPEN ; channel: summary_demux port 1: (cntrl) - ; channel: $

[process] dada_single status=ACTIVE port 0: (queue) OPEN ; channel: demux port 1: (queue) OPEN ; channel: trunc port 2: (queue) OPEN ; channel: MATPLOTLIBRC port 3: (cntrl) - ; channel: $

[process] classifier status=ACTIVE port 0: (queue) OPEN ; channel: repseq port 1: (queue) OPEN ; channel: trained_classifier port 2: (queue) OPEN ; channel: MATPLOTLIBRC port 3: (cntrl) - ; channel: $

[process] filter_taxa status=ACTIVE port 0: (queue) OPEN ; channel: table port 1: (queue) OPEN ; channel: repseq port 2: (queue) OPEN ; channel: taxonomy port 3: (queue) OPEN ; channel: MATPLOTLIBRC port 4: (cntrl) - ; channel: $

[process] export_filtered_dada_output status=ACTIVE port 0: (queue) OPEN ; channel: table port 1: (queue) OPEN ; channel: repseq port 2: (queue) OPEN ; channel: MATPLOTLIBRC port 3: (cntrl) - ; channel: $

[process] report_filter_stats status=ACTIVE port 0: (queue) OPEN ; channel: unfiltered_table port 1: (queue) OPEN ; channel: filtered_table port 2: (cntrl) - ; channel: $

[process] RelativeAbundanceASV status=ACTIVE port 0: (queue) OPEN ; channel: table port 1: (queue) OPEN ; channel: MATPLOTLIBRC port 2: (cntrl) - ; channel: $

[process] RelativeAbundanceReducedTaxa status=ACTIVE port 0: (queue) OPEN ; channel: table port 1: (queue) OPEN ; channel: taxonomy port 2: (queue) OPEN ; channel: MATPLOTLIBRC port 3: (cntrl) - ; channel: $

[process] barplot status=ACTIVE port 0: (queue) OPEN ; channel: metadata port 1: (queue) OPEN ; channel: table port 2: (queue) OPEN ; channel: taxonomy port 3: (queue) OPEN ; channel: MATPLOTLIBRC port 4: (cntrl) - ; channel: $

[process] tree status=ACTIVE port 0: (queue) OPEN ; channel: repseq port 1: (queue) OPEN ; channel: MATPLOTLIBRC port 2: (cntrl) - ; channel: $

[process] alpha_rarefaction status=ACTIVE port 0: (queue) OPEN ; channel: metadata port 1: (queue) OPEN ; channel: table port 2: (queue) OPEN ; channel: tree port 3: (queue) OPEN ; channel: stats port 4: (queue) OPEN ; channel: MATPLOTLIBRC port 5: (cntrl) - ; channel: $

[process] combinetable status=ACTIVE port 0: (queue) OPEN ; channel: TABLE port 1: (queue) OPEN ; channel: SEQ port 2: (queue) OPEN ; channel: TAXONOMY port 3: (cntrl) - ; channel: $

[process] diversity_core status=ACTIVE port 0: (queue) OPEN ; channel: metadata port 1: (queue) OPEN ; channel: table port 2: (queue) OPEN ; channel: tree port 3: (queue) OPEN ; channel: stats port 4: (queue) OPEN ; channel: MATPLOTLIBRC port 5: (cntrl) - ; channel: $

[process] metadata_category_all status=ACTIVE port 0: (queue) closed; channel: metadata port 1: (queue) OPEN ; channel: MATPLOTLIBRC port 2: (cntrl) - ; channel: $

[process] metadata_category_pairwise status=ACTIVE port 0: (queue) closed; channel: metadata port 1: (queue) OPEN ; channel: MATPLOTLIBRC port 2: (cntrl) - ; channel: $

[process] alpha_diversity status=ACTIVE port 0: (queue) OPEN ; channel: - port 1: (cntrl) - ; channel: $

[process] beta_diversity status=ACTIVE port 0: (queue) OPEN ; channel: - port 1: (cntrl) - ; channel: $

[process] beta_diversity_ordination status=ACTIVE port 0: (queue) OPEN ; channel: - port 1: (cntrl) - ; channel: $

[process] prepare_ancom status=ACTIVE port 0: (queue) OPEN ; channel: metadata port 1: (queue) OPEN ; channel: table port 2: (queue) OPEN ; channel: meta port 3: (queue) OPEN ; channel: MATPLOTLIBRC port 4: (cntrl) - ; channel: $

[process] ancom_tax status=ACTIVE port 0: (queue) OPEN ; channel: - port 1: (cntrl) - ; channel: $

[process] ancom_asv status=ACTIVE port 0: (queue) OPEN ; channel: - port 1: (cntrl) - ; channel: $

[process] output_documentation status=ACTIVE port 0: (queue) closed; channel: output_docs port 1: (cntrl) - ; channel: $

Jul-29 11:55:25.983 [Actor Thread 14] DEBUG nextflow.container.SingularityCache - Singularity pull complete image=docker://nfcore/ampliseq:1.0.0 path=/media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/singularity/nfcore-ampliseq-1.0.0.img Jul-29 11:55:26.119 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Jul-29 11:55:26.129 [Task submitter] INFO nextflow.Session - [95/26f20b] Submitted process > output_documentation (1) Jul-29 11:55:26.145 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Jul-29 11:55:26.149 [Task submitter] INFO nextflow.Session - [fb/e209a0] Submitted process > fastqc (2_S115) Jul-29 11:55:26.231 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Jul-29 11:55:26.234 [Task submitter] INFO nextflow.Session - [8b/3a7eb5] Submitted process > fastqc (1a_S103) Jul-29 11:55:26.242 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Jul-29 11:55:26.245 [Task submitter] INFO nextflow.Session - [3e/6a89b3] Submitted process > metadata_category_pairwise (1) Jul-29 11:55:28.954 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: fastqc (1a_S103); status: COMPLETED; exit: 127; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/8b/3a7eb5e9f10f2353893c02a350babc] Jul-29 11:55:28.964 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Jul-29 11:55:28.964 [Task submitter] INFO nextflow.Session - [a2/36dc81] Submitted process > fastqc (1_S103) Jul-29 11:55:28.983 [Task monitor] INFO nextflow.processor.TaskProcessor - [8b/3a7eb5] NOTE: Process fastqc (1a_S103) terminated with an error exit status (127) -- Error is ignored Jul-29 11:55:29.026 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 11; name: metadata_category_pairwise (1); status: COMPLETED; exit: 127; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/3e/6a89b32e8ac3b6a9d8a4e4fa482f94] Jul-29 11:55:29.028 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Jul-29 11:55:29.029 [Task submitter] INFO nextflow.Session - [ed/5be743] Submitted process > metadata_category_all (1) Jul-29 11:55:29.086 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'metadata_category_pairwise (1)'

Caused by: Process metadata_category_pairwise (1) terminated with an error exit status (127)

Command executed:

metadataCategoryPairwise.r Metadata.tsv

Command exit status: 127

Command output: (empty)

Command error: /bin/bash: line 0: cd: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/3e/6a89b32e8ac3b6a9d8a4e4fa482f94: No such file or directory /bin/bash: .command.run: No such file or directory

Work dir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/3e/6a89b32e8ac3b6a9d8a4e4fa482f94

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out Jul-29 11:55:29.109 [Task monitor] INFO nextflow.Session - Execution cancelled -- Finishing pending tasks before exit Jul-29 11:55:29.121 [main] DEBUG nextflow.Session - Session await > all process finished Jul-29 11:55:29.129 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 12; name: output_documentation (1); status: COMPLETED; exit: 127; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/95/26f20b3cb0e9e0884fa73cca639fa8] Jul-29 11:55:29.131 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: fastqc (2_S115); status: COMPLETED; exit: 127; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/fb/e209a0b5991d131d4208691b35a584] Jul-29 11:55:29.132 [Task monitor] INFO nextflow.processor.TaskProcessor - [fb/e209a0] NOTE: Process fastqc (2_S115) terminated with an error exit status (127) -- Error is ignored Jul-29 11:55:29.448 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: fastqc (1_S103); status: COMPLETED; exit: 127; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/a2/36dc8106b75a0546e6ec467a90159a] Jul-29 11:55:29.449 [Task monitor] INFO nextflow.processor.TaskProcessor - [a2/36dc81] NOTE: Process fastqc (1_S103) terminated with an error exit status (127) -- Error is ignored Jul-29 11:55:29.523 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 10; name: metadata_category_all (1); status: COMPLETED; exit: 127; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/ed/5be7434fb10cd7af77a63d12cf6874] Jul-29 11:57:29.358 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled Jul-29 12:00:26.617 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 6 -- tasks to be submitted are shown below ~> TaskHandler[id: 8; name: trimming (1a_S103); status: NEW; exit: -; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/9c/75ed76fcd7c6e8fc2d90c60a1051c1] ~> TaskHandler[id: 9; name: trimming (2a_S115); status: NEW; exit: -; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/28/2640df3b3ce8db5d5590cc0522ffc1] ~> TaskHandler[id: 7; name: trimming (2_S115); status: NEW; exit: -; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/fa/e134eff77f7d89c7f5f9301cceb382] ~> TaskHandler[id: 6; name: trimming (1_S103); status: NEW; exit: -; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/8f/8775f1cbccf72c876e7a92582a24ff] ~> TaskHandler[id: 5; name: fastqc (2a_S115); status: NEW; exit: -; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/74/56b08da568a0c05a432d5385bca231] ~> TaskHandler[id: 1; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /media/ilias/STORAGE/Sandra/nf-core/singularit_test/work/09/3127ab90fffdf1f946820a047e2b33]

d4straub commented 4 years ago

Well, this time the container pull is successful, congratulation!

Other than that, it seems to me that the pipeline isn't able to write in the work directory. This could be access right problems or connection problems (after all it seems a different partition: /media/).

MonicaSteffi commented 4 years ago

Other than that, it seems to me that the pipeline isn't able to write in the work directory. This could be access right problems or connection problems (after all it seems a different partition: /media/).

Hi @d4straub I checked the permission of my folder. It has writing access to. Yeah. It is a different partition and how do I execute in this?

d4straub commented 4 years ago

Theoretically there should not be any problem. But practically you obviously do have trouble. Would you be able to try it on another partition/hardware?

MonicaSteffi commented 4 years ago

@d4straub I am running it on desktop. Its working fine except the memory error. ERROR ~ Error executing process > 'data_single'

Caused by: Process requirement exceed available CPUs -- req: 8; avail: 4

So I tried with the solution mention in the post as [(https://github.com/nf-core/rnaseq/issues/130)]. Created a config file as follows

process {
  withName:dada_single {
    cpus = 4
  }
withName:classifier{
    cpus = 4
  }
}

But apart from this error, I think it is working fine in Desktop but not on media partition

d4straub commented 4 years ago

While the solution with the config file is fine, it would be probably easier to use --max_cpus 4 see here.

I do not know why it doesn't work on the partition, but it might be not a problem with ampliseq since it does work on your Desktop. Unfortunately I do not see another way than not using that partition and instead use other hardware.

MonicaSteffi commented 4 years ago

While the solution with the config file is fine, it would be probably easier to use --max_cpus 4 see here.

I do not know why it doesn't work on the partition, but it might be not a problem with ampliseq since it does work on your Desktop. Unfortunately I do not see another way than not using that partition and instead use other hardware.

thank you.