search

epi2me-labs / wf-16s

Other
22 stars 5 forks source link

prepare_database fails #17

Closed hanburglarbabel closed 5 months ago

hanburglarbabel commented 6 months ago

Operating System

Other Linux (please specify below)

Other Linux

Ubuntu 20.04.6 LTS

Workflow Version

v1.1.3

Workflow Execution

EPI2ME Desktop (Local)

Other workflow execution

command line (local)

EPI2ME Version

v5.1.10

CLI command run

nextflow run epi2me-labs/wf-16s --fastq wf-16s-demo/test_data/ -profile standard

Workflow Execution - CLI Execution Profile

standard (default)

What happened?

Both on the desktop application and command line I am running into errors with the prepare_database. I have tried with all 3 databases as well as gave it a shot with local silva138 (however I didnt have a ref2taxid file, so quickly gave up on that). The process keeps failing with a command exit status: 4 and issues it seems with the ncbi host address. I have tried both with my own data, as well as with the demo and the same issue is coming up with both, so here is the output from the demo.

Relevant log output

N E X T F L O W   ~  version 24.04.1

Launching `epi2me-labs/wf-16s/main.nf` [prickly_montalcini] DSL2 - revision: 8363748886

WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES

||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-16s v1.1.3
--------------------------------------------------------------------------------
Core Nextflow options
  runName        : prickly_montalcini
  containerEngine: docker
  container      : [withLabel:wfmetagenomics:ontresearch/wf-metagenomics:sha44a6dacff5f2001d917b774647bb4cbc1b53bc76, withLabel:wf_common:ontresearch/wf-common:sha645176f98b8780851f9c476a064d44c2ae76ddf6]
  launchDir      : /home/hba052/epi2melabs/workflows
  workDir        : /home/hba052/epi2melabs/workflows/work
  projectDir     : /home/hba052/epi2melabs/workflows/epi2me-labs/wf-16s
  userName       : hba052
  profile        : standard
  configFiles    : /home/hba052/epi2melabs/workflows/epi2me-labs/wf-16s/nextflow.config

Input Options
  fastq          : wf-16s-demo/test_data/

Reference Options
  database_sets  : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], SILVA_138_1:[database:null]]

!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-16s for your analysis please cite:

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

--------------------------------------------------------------------------------
This is epi2me-labs/wf-16s v1.1.3.
--------------------------------------------------------------------------------
Checking inputs.
Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
[-        ] fastcat -
Note: Empty files or those files whose reads have been discarded after filtering[-        ] fastcat                        -
[-        ] pre…s:download_unpack_taxonomy -
executor >  local (3)
[e1/e576a5] fastcat (1)                    | 0 of 3
executor >  local (5)
[e1/e576a5] fastcat (1)                    | 0 of 3
executor >  local (5)
[e1/e576a5] fastcat (1)                    | 0 of 3
executor >  local (7)
[1e/35b1ba] fastcat (3)                    | 1 of 3
executor >  local (7)
[1e/35b1ba] fastcat (3)                    | 1 of 3
executor >  local (7)
[1e/35b1ba] fastcat (3)                    | 1 of 3
executor >  local (8)
[ab/d11085] fastcat (2)                    | 2 of 3
executor >  local (9)
[ab/d11085] fastcat (2)                    | 2 of 3
executor >  local (9)
[ab/d11085] fastcat (2)                    | 2 of 3
executor >  local (9)
[ab/d11085] fastcat (2)                    | 3 of 3 ✔
executor >  local (9)
[ab/d11085] fastcat (2)                    | 3 of 3 ✔
[71/19ea3d] pre…s:download_unpack_taxonomy | 0 of 1
[28/8eddec] pre…wnload_reference_ref2taxid | 0 of 1
[18/e0e10d] min…ine:run_common:getVersions | 1 of 1 ✔
[04/cce01c] min…eline:run_common:getParams | 1 of 1 ✔
[-        ] minimap_pipeline:minimap       -
[-        ] min…line:createAbundanceTables -
[-        ] minimap_pipeline:makeReport    -
[8b/b4d14f] min…ipeline:output_results (2) | 2 of 2

ERROR ~ Error executing process > 'prepare_databases:download_unpack_taxonomy'

Caused by:
  Process `prepare_databases:download_unpack_taxonomy` terminated with an error exit status (4)

Command executed:

  # Check if the folder is an url to fetch or a local path 
  if true
  then
      wget 'https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip'
  fi
  # Uncompress db
  if [[ "taxdmp_2023-01-01.zip" == *.tar.gz ]]
      then
          mkdir taxdmp_2023-01-01_db
          tar xf "taxdmp_2023-01-01.zip" -C taxdmp_2023-01-01_db
  elif [[ "taxdmp_2023-01-01.zip" == *.zip ]]
      then
          mkdir taxdmp_2023-01-01_db
          unzip  "taxdmp_2023-01-01.zip" -d taxdmp_2023-01-01_db
  else
          echo "Error: taxonomy is neither .tar.gz, .zip"
          echo "Exiting".
          exit 1
  fi

Command exit status:
  4

executor >  local (9)
[ab/d11085] fastcat (2)                    | 3 of 3 ✔
[71/19ea3d] pre…s:download_unpack_taxonomy | 1 of 1, failed: 1 ✘
[28/8eddec] pre…wnload_reference_ref2taxid | 0 of 1
[18/e0e10d] min…ine:run_common:getVersions | 1 of 1 ✔
[04/cce01c] min…eline:run_common:getParams | 1 of 1 ✔
[-        ] minimap_pipeline:minimap       -
[-        ] min…line:createAbundanceTables -
[-        ] minimap_pipeline:makeReport    -
[8b/b4d14f] min…ipeline:output_results (2) | 2 of 2 ✔
ERROR ~ Error executing process > 'prepare_databases:download_unpack_taxonomy'

Caused by:
  Process `prepare_databases:download_unpack_taxonomy` terminated with an error exit status (4)

Command executed:

  # Check if the folder is an url to fetch or a local path 
  if true
  then
      wget 'https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip'
  fi
  # Uncompress db
  if [[ "taxdmp_2023-01-01.zip" == *.tar.gz ]]
      then
          mkdir taxdmp_2023-01-01_db
          tar xf "taxdmp_2023-01-01.zip" -C taxdmp_2023-01-01_db
  elif [[ "taxdmp_2023-01-01.zip" == *.zip ]]
      then
          mkdir taxdmp_2023-01-01_db
          unzip  "taxdmp_2023-01-01.zip" -d taxdmp_2023-01-01_db
  else
          echo "Error: taxonomy is neither .tar.gz, .zip"
          echo "Exiting".
          exit 1
  fi

Command exit status:
  4

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
executor >  local (9)
[ab/d11085] fastcat (2)                    | 3 of 3 ✔
[71/19ea3d] pre…s:download_unpack_taxonomy | 1 of 1, failed: 1 ✘
[-        ] pre…wnload_reference_ref2taxid -
[18/e0e10d] min…ine:run_common:getVersions | 1 of 1 ✔
[04/cce01c] min…eline:run_common:getParams | 1 of 1 ✔
[-        ] minimap_pipeline:minimap       -
[-        ] min…line:createAbundanceTables -
[-        ] minimap_pipeline:makeReport    -
[8b/b4d14f] min…ipeline:output_results (2) | 2 of 2 ✔
ERROR ~ Error executing process > 'prepare_databases:download_unpack_taxonomy'

Caused by:
  Process `prepare_databases:download_unpack_taxonomy` terminated with an error exit status (4)

Command executed:

  # Check if the folder is an url to fetch or a local path 
  if true
  then
      wget 'https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip'
  fi
  # Uncompress db
  if [[ "taxdmp_2023-01-01.zip" == *.tar.gz ]]
      then
          mkdir taxdmp_2023-01-01_db
          tar xf "taxdmp_2023-01-01.zip" -C taxdmp_2023-01-01_db
  elif [[ "taxdmp_2023-01-01.zip" == *.zip ]]
      then
          mkdir taxdmp_2023-01-01_db
          unzip  "taxdmp_2023-01-01.zip" -d taxdmp_2023-01-01_db
  else
          echo "Error: taxonomy is neither .tar.gz, .zip"
          echo "Exiting".
          exit 1
  fi

Command exit status:
  4

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  --2024-05-22 12:57:25--  https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip
  Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... failed: Temporary failure in name resolution.
  wget: unable to resolve host address 'ftp.ncbi.nlm.nih.gov'
  cp: '.command.out' and '.command.out' are the same file
  cp: '.command.err' and '.command.err' are the same file
  cp: '.command.trace' and '.command.trace' are the same file

Work dir:
  /home/hba052/epi2melabs/workflows/work/71/19ea3df393d90eb49dee4d56b9e190

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

Application activity log entry

No red items

Were you able to successfully run the latest version of the workflow with the demo data?

no

Other demo data information

No response

nggvs commented 5 months ago

Hi @hanburglarbabel , As you have mentioned, it seems that you are having problems connecting with the NCBI FTP site. Unfortunately, I am not able to reproduce it, could you run wget 'https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip' or just navigate to it in the explorer?

nggvs commented 5 months ago

Hi @hanburglarbabel , I'll close the issue as I have no more news about it, please feel free to reopen it (or open a new one if it is something different) if it still doesn't work. Thank you very much for using the workflow!