search

epi2me-labs / wf-flu

Other
17 stars 5 forks source link

Unable to run demo-data, and without clue of any error in the .nexflow.log file #11

Open MauriAndresMU1313 opened 1 month ago

MauriAndresMU1313 commented 1 month ago

Operating System

Ubuntu 22.04

Other Linux

No response

Workflow Version

wf-flu v1.2.3-g5290f79

Workflow Execution

Command line (Cluster)

Other workflow execution

No response

EPI2ME Version

No response

CLI command run

nextflow run epi2me-labs/wf-flu --downsample 50 --fastq 'demo_data/wf-flu-demo/fastq/' --sample_sheet 'demo_data/wf-flu-demo/sample_sheet.csv' -profile standard

Workflow Execution - CLI Execution Profile

None

What happened?

I was trying to run the demo-data to check if the pipeline is working properly. Previously, the pipeline ran until an specific step, but was running, I would like to remark that.

Relevant log output

Oct-21 15:39:37.015 [main] DEBUG nextflow.cli.Launcher - $> nextflow run epi2me-labs/wf-flu --downsample 50 --fastq demo_data/wf-flu-demo/fastq/ --sample_sheet demo_data/wf-flu-demo/sample_sheet.csv -profile standard
Oct-21 15:39:37.138 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.2
Oct-21 15:39:37.170 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/mmora30/.nextflow/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2,nf-tower@1.9.1,nf-wave@1.4.2
Oct-21 15:39:37.187 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Oct-21 15:39:37.189 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Oct-21 15:39:37.194 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Oct-21 15:39:37.216 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Oct-21 15:39:37.236 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /home/mmora30/.nextflow/scm
Oct-21 15:39:38.257 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-flu.git
Oct-21 15:39:38.302 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Oct-21 15:39:38.322 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-flu.git
Oct-21 15:39:39.112 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/nextflow.config
Oct-21 15:39:39.116 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/nextflow.config
Oct-21 15:39:39.129 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/mmora30/.nextflow/secrets/store.json
Oct-21 15:39:39.133 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@18a096b5] - activable => nextflow.secret.LocalSecretsProvider@18a096b5
Oct-21 15:39:39.148 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Oct-21 15:39:39.467 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Oct-21 15:39:39.469 [main] DEBUG nextflow.cli.CmdRun - Launching `https://github.com/epi2me-labs/wf-flu` [sad_rosalind] DSL2 - revision: 5290f79553 [master]
Oct-21 15:39:39.471 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Oct-21 15:39:39.472 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Oct-21 15:39:39.543 [main] DEBUG nextflow.Session - Session UUID: 582afe70-4d0d-45c1-96ed-90aaf9eb7b17
Oct-21 15:39:39.543 [main] DEBUG nextflow.Session - Run name: sad_rosalind
Oct-21 15:39:39.544 [main] DEBUG nextflow.Session - Executor pool size: 32
Oct-21 15:39:39.554 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Oct-21 15:39:39.561 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Oct-21 15:39:39.585 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC
  System: Linux 5.15.0-119-generic
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.24+8-post-Ubuntu-1ubuntu322.04
  Encoding: UTF-8 (UTF-8)
  Process: 1864037@medina-lab [127.0.1.1]
  CPUs: 32 - Mem: 251.6 GB (212.3 GB) - Swap: 8 GB (8 GB)
Oct-21 15:39:39.612 [main] DEBUG nextflow.Session - Work-dir: /localData/workspace_mm/nextflow_pipelines/epi2me-labs/wf-flu/work [ext2/ext3]
Oct-21 15:39:39.678 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Oct-21 15:39:39.699 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Oct-21 15:39:39.790 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Oct-21 15:39:39.806 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 33; maxThreads: 1000
Oct-21 15:39:39.959 [main] DEBUG nextflow.Session - Session start
Oct-21 15:39:39.965 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /localData/workspace_mm/nextflow_pipelines/epi2me-labs/wf-flu/output/execution/trace.txt
Oct-21 15:39:39.976 [main] DEBUG nextflow.Session - Using default localLib path: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/lib
Oct-21 15:39:39.981 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/lib
Oct-21 15:39:39.982 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/lib/nfcore_external_java_deps.jar
Oct-21 15:39:42.619 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Oct-21 15:39:46.417 [main] INFO  nextflow.Nextflow -
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-flu v1.2.3-g5290f79
--------------------------------------------------------------------------------
Core Nextflow options
  revision       : master
  runName        : sad_rosalind
  containerEngine: docker
  container      : [withLabel:wf_common:ontresearch/wf-common:shad28e55140f75a68f59bbecc74e880aeab16ab158, withLabel:wfflu:ontresearch/wf-flu:sha5eeb2cd0a75a6cc0f47d689d2926e0d9922ceda8, withLabel:wfflutyping:ontresearch/abricate:sha671198c26ad22c97e3cdaa81a646ec0c215ebc43, withLabel:nextclade:ontresearch/nextclade:shab67652b34e898ec8dea18d0953a0b06b10466f76, withLabel:medaka:ontresearch/medaka:sha3486abaab0d3b90351617eb8622acf2028edb154]
  launchDir      : /localData/workspace_mm/nextflow_pipelines/epi2me-labs/wf-flu
  workDir        : /localData/workspace_mm/nextflow_pipelines/epi2me-labs/wf-flu/work
  projectDir     : /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu
  userName       : mmora30
  profile        : standard
  configFiles    : /home/mmora30/.nextflow/assets/epi2me-labs/wf-flu/nextflow.config

Input Options
  fastq          : demo_data/wf-flu-demo/fastq/

Sample Options
  sample_sheet   : demo_data/wf-flu-demo/sample_sheet.csv

Advanced Options
  downsample     : 50

!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-flu for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

--------------------------------------------------------------------------------
This is epi2me-labs/wf-flu v1.2.3-g5290f79.
--------------------------------------------------------------------------------
Oct-21 15:39:47.038 [main] INFO  nextflow.Nextflow - Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
Oct-21 15:39:47.176 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wf_common` matches labels `ingress,wf_common` for process with name validate_sample_sheet
Oct-21 15:39:47.204 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-21 15:39:47.205 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-21 15:39:47.212 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Oct-21 15:39:47.220 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=32; memory=251.7 GB; capacity=32; pollInterval=100ms; dumpInterval=5m
Oct-21 15:39:47.223 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

other (please describe below)

Other demo data information

I was able to ran until Medaka step, where I got error related to AVX instructions. However, now I do not have any error associate to this last one.
MauriAndresMU1313 commented 1 month ago

I would like to add to the post that when I close the session, the workflow stop, when the fact is that I was running in the background and the output show this:

--------------------------------------------------------------------------------
This is epi2me-labs/wf-flu v1.2.3-g5290f79.
--------------------------------------------------------------------------------
Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
[-        ] validate_sample_sheet -

[-        ] validate_sample_sheet -
[-        ] fastcat               -
[-        ] pipeline:alignReads   -
[-        ] pipeline:coverageCalc -
Downsampling!!!

[-        ] validate_sample_sheet      -
[-        ] fastcat                    -
[-        ] pipeline:alignReads        -
[-        ] pipeline:coverageCalc      -
[-        ] pipeline:downSample        -
[-        ] pipeline:medakaVariants    -
[-        ] pipeline:makeConsensus     -
[-        ] pipeline:typeFlu           -
[-        ] pipeline:processType       -
[-        ] pipeline:prepNextclade     -
[-        ] pipeline:nextclade         -
[-        ] pipeline:getVersions       -
[-        ] pipeline:getParams         -
[-        ] pipeline:collectFilesInDir -
Downsampling!!!

[-        ] validate_sample_sheet      -
[-        ] fastcat                    -
[-        ] pipeline:alignReads        -
[-        ] pipeline:coverageCalc      -
[-        ] pipeline:downSample        -
[-        ] pipeline:medakaVariants    -
[-        ] pipeline:makeConsensus     -
[-        ] pipeline:typeFlu           -
[-        ] pipeline:processType       -
[-        ] pipeline:prepNextclade     -
[-        ] pipeline:nextclade         -
[-        ] pipeline:getVersions       -
[-        ] pipeline:getParams         -
[-        ] pipeline:collectFilesInDir -
[-        ] pipeline:makeReport        -
[-        ] output                     -
Downsampling!!!

And the .log file shows the interrupted error:

Oct-21 17:53:23.676 [Actor Thread 31] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=validate_sample_sheet; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Oct-21 17:53:23.676 [Actor Thread 32] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:getParams; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Oct-21 17:53:23.677 [Actor Thread 25] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=pipeline:getVersions; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException

So, looks like the pipeline stay active without run any process

cjalder commented 1 month ago

Hi @MauriAndresMU1313 - how long had you been running the workflow before you closed the session? It could be that the workflow was downloading the required docker images before any of the processes began?

MauriAndresMU1313 commented 1 month ago

Thank you for your quick response! I had to leave the session open for at least 4 hours. Even so, I run the following command to run in the background:

nohup bash command_instructions.sh &

This worked before, and in a previous post I mentioned that the pipeline in my VM stopped at the medaka step, so since I installed the conda environment with Tensor Flow with AVX instructions, the pipeline went down. I know that does not make sense, because it is a conda environment, so I tried with the conda activate and deactivate but as I mentioned here, the pipeline does nothing. Maybe download the docker image again? Any suggestion?