search

replikation / What_the_Phage

WtP: Phage identification via nextflow and docker or singularity
https://mult1fractal.github.io/wtp-documentation/
GNU General Public License v3.0
102 stars 15 forks source link

Errors running WtP #171

Closed dinahoch closed 2 years ago

dinahoch commented 2 years ago

Hi,

I've been trying to run WtP for a couple of days now and keep encountering errors that have been challenging to resolve. The latest is:

Error executing process > 'pvog_database:pvog_DB'

Caused by:
  Process `pvog_database:pvog_DB` terminated with an error exit status (1)

Command executed:

  wget -nH ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz && tar -zxvf pvogs.tar.gz
  rm pvogs.tar.gz

Command exit status:
  1

Command output:
  (empty)

Command error:
  --2022-08-03 14:06:14--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:14) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Retrying.

  --2022-08-03 14:06:28--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:15) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Retrying.

  --2022-08-03 14:06:42--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:16) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Retrying.

  --2022-08-03 14:06:56--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:17) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Retrying.

  --2022-08-03 14:07:10--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:18) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Retrying.

  --2022-08-03 14:07:25--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:19) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Retrying.

  --2022-08-03 14:07:39--  ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/viral-pipeline/hmmer_databases/pvogs.tar.gz
    (try:20) => ‘pvogs.tar.gz’
  Connecting to ftp.ebi.ac.uk (ftp.ebi.ac.uk)|193.62.193.138|:21... connected.
  Logging in as anonymous ...
  Error in server response, closing control connection.
  Giving up.

  rm: cannot remove 'pvogs.tar.gz': No such file or directory

Work dir:
  /home/labs/sorek/dinah/WhatThePhage/work/a0/ac622ce65c592768d7565107452012

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

I am running the code nextflow run replikation/What_the_Phage -r v1.0.1 --cores 8 -profile smalltest,singularity --database WtP_Nextflow_databases_unzip/ using an unzipped database that I downloaded from here: https://osf.io/wtfrc/

It seems that the code is not using the WtP_Nextflow_databases_unzip/vog/vogdb file for some reason and so trying to download it and failing. I would appreciate any help on how I could resolve this issue.

Thank you in advance

mult1fractal commented 2 years ago

Hey :)

could you please run -r v1.1.0

in the version you use I had not yet implemented the failsave for database downloads that you are encountering atm.

Also could you delete the WtP_Nextflow_databases_unzip folder to make sure the proper databases will be used/downloaded

replikation commented 2 years ago

might also be some firewall issue? i mean its the Ebi it should be reachable

mult1fractal commented 2 years ago

mhh... I think the pvogs download page is down

replikation commented 2 years ago

yeah but its stored at ftp://ftp.ebi.ac.uk

dinahoch commented 2 years ago

Thank you for your suggestions :)

I am now running the command nextflow run replikation/What_the_Phage -r v1.1.0 --setup -profile local,singularity and getting the error message:

[93/800140] NOTE: Missing output file(s) `pvogs` expected by process `setup_wf:pvog_DB` -- Execution is retried (1)
Error executing process > 'setup_wf:pvog_DB'

Caused by:
  Process `setup_wf:pvog_DB` terminated with an error exit status (1)

Command executed:

  https://osf.io/n6ph5/download -O pvog.tar.gz
  tar -xvzf pvog.tar.gz
  cp -r pvog/pvogs/ .
  rm -r pvog
  rm pvog.tar.gz

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.sh: line 2: https://osf.io/n6ph5/download: No such file or directory

Work dir:
  /tmp/nextflow-phages-dinah/54/ab99341875a15cf373019070ff4b55

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

WARN: Killing running tasks (4)

I’m not sure what the problem is, but I do see that in the phage_deprecated.nf script, it directs to "${params.databases}/pvogs/", while the relevant directory seems to be WtP_Nextflow_databases/pvog/pvogs.

mult1fractal commented 2 years ago

hmm ok thats weird... I think nextflow is not using the most recent scripts The command looks also not fully correct.. this is the most recent command from v1.1.0:

 wget --no-check-certificate https://osf.io/n6ph5/download -O pvog.tar.gz
        tar -xvzf pvog.tar.gz
        cp -r pvog/pvogs/ .
        rm -r pvog
        rm pvog.tar.gz
        """

and the file is stored here: "${params.databases}/pvog"

could you delete the pvog folder in your WtP_Nextflow_databases folder? and then restart the workflow

if this also does not work: delete all WtP related folders (also in /tmp/) and redo nextflow run replikation/What_the_Phage -r v1.1.0 --setup -profile local,singularity I guess nextflow still uses the old v1.0.1 scripts

dinahoch commented 2 years ago 
N E X T F L O W  ~  version 22.08.0-edge
Launching `https://github.com/replikation/What_the_Phage` [lethal_monod] DSL2 - revision: 1e53b61915 [v1.1.0]
_____ _____ ____ ____ ___ ___ __ __ _ _
  __      _______________________
 /  \    /  \__    ___/\______   \
 \   \/\/   / |    |    |     ___/
  \        /  |    |    |    |
   \__/\  /   |____|    |____|
        \/
_____ _____ ____ ____ ___ ___ __ __ _ _

Profile: local,singularity

Current User: dinah
Nextflow-version: 22.08.0.edge
Starting time: 01-08-2022 16:24 UTC
Workdir location [--workdir]:
  /tmp/nextflow-phages-dinah
Output location [--output]:
  results
Database location [--databases]:
  nextflow-autodownload-databases
Singularity cache location [--cachedir]:
  singularity_images

  WARNING: Singularity image building sometimes fails!
  Please download all images first via --setup --cachedir IMAGE-LOCATION
  Manually remove faulty images in singularity_images for a rebuild

CPUs per process: 10, maximal CPUs to use: 40

executor >  local (12)
executor >  local (12)
[-        ] process > setup_wf:setup_container (1)       -
[fd/cae9a3] process > setup_wf:download_references       [100%] 1 of 1 ✔
[42/075fed] process > setup_wf:phage_references_blastDB  [100%] 1 of 1 ✔
[e8/701d60] process > setup_wf:ppr_download_dependencies [100%] 1 of 1 ✔
[aa/160b1b] process > setup_wf:sourmash_download_DB      [100%] 1 of 1 ✔
[-        ] process > setup_wf:vibrant_download_DB       -
[-        ] process > setup_wf:virsorter_download_DB     -
[0c/8cbd07] process > setup_wf:virsorter2_download_DB    [100%] 1 of 1 ✔
[c3/df9d19] process > setup_wf:pvog_DB                   [100%] 2 of 2, failed: 2, retries: 1 ✘
[d5/fb0f5a] process > setup_wf:vogtable_DB               [100%] 1 of 1 ✔
[c8/f21847] process > setup_wf:download_checkV_DB        [100%] 1 of 1 ✔
[03/9f42f4] NOTE: Missing output file(s) `pvogs` expected by process `setup_wf:pvog_DB` -- Execution is retried (1)
Error executing process > 'setup_wf:pvog_DB'

Caused by:
  Process `setup_wf:pvog_DB` terminated with an error exit status (1)

Command executed:

  https://osf.io/n6ph5/download -O pvog.tar.gz
  tar -xvzf pvog.tar.gz
  cp -r pvog/pvogs/ .
  rm -r pvog
  rm pvog.tar.gz

Command exit status:
  1

Command output:
  (empty)

Command error:
  .command.sh: line 2: https://osf.io/n6ph5/download: No such file or directory

Work dir:
  /tmp/nextflow-phages-dinah/c3/df9d19f49aa53708516f4eb8f0bb77

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Still no luck... This was after deleting all files and running the command nextflow run replikation/What_the_Phage -r v1.1.0 --setup -profile local,singularity

mult1fractal commented 2 years ago

ok, mhh, I'm not a singularity expert as I only use docker what I could think of that there are still some "wrong singularity-images" that are being used from your first setup try (v1.0.1) could you remove the "old singularity images"

and execute this command nextflow run replikation/What_the_Phage -r v1.1.0 --setup -profile local,singularity with --cachedir IMAGE-LOCATION as described here: WARNING: Singularity image building sometimes fails! Please download all images first via --setup --cachedir IMAGE-LOCATION Manually remove faulty images in singularity_images for a rebuild

this should build/use the new singularity images from v 1.1.0

I hope that works

yesterday I deleted all my databases and run your command only with -profile local,docker and I had no issues... so I guess it is because of the "old singularity images"

dinahoch commented 2 years ago

😢 Still not working. If you have a bit of time to help me directly I would really appreciate it. I've emailed you!

mult1fractal commented 2 years ago

solved via google meeting

dinahoch commented 2 years ago

In case anyone has a similar issue, this was resolved by cloning the github depository and running the script directly using the commands

git clone https://github.com/replikation/What_the_Phage.git
cd What_the_Phage
conda activate wtp
nextflow run phage.nf --setup -profile local,singularity --cachedir singularity_images/

Thank you Mike for taking the time to help! 😄