search

vinisalazar / metaphor

Metaphor: a general-purpose workflow for assembly and binning of metagenomes
https://metaphor-workflow.readthedocs.io/
Other
35 stars 3 forks source link

Error while running "metaphor test" command line #51

Closed BioInfoStar5 closed 11 months ago

BioInfoStar5 commented 11 months ago

Dear colleagues, I got this error while running "metaphor test" command line for metagenomics data analysis. The workflow started correctly and data has been downloaded properly. However, the workflow stopped and terminal shows this error:

File "/miniconda2/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', '/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/test-config.yaml', '--cores', '3', '-p', '-r', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda', '--config', 'max_mb=8192']' returned non-zero exit status 1.

Do you have any idea how to resolve this error Many thanks

vinisalazar commented 11 months ago

Hello @BioInfoStar5,

Thank you for opening an issue. It is hard to identify the problem from this error message alone, could you please provide more context? Are you running Metaphor directly on the command-line or as a script? Could you post a screenshot or paste the command that you've run, followed by the Terminal output?

Best, Vini

BioInfoStar5 commented 11 months ago

Thanks Vini, Yes I'm running Metaphor directly on the command-line. The command line I entred was: $ metaphor test Here, you will find the screenshots of terminal output: Page1 Page2

Many thanks

vinisalazar commented 11 months ago

Hello @BioInfoStar5, thank you for clarifying.

Your screenshot shows:

PermissionError: [Errno 13] Permission denied: 'home/inra/.cache/snakemake/snakemake/source-cache/runtime-cache'

That indicates a problem related to the interaction between your OS and the Snakemake installation. This issue seems to discuss it: snakemake/snakemake#1593, perhaps try some of the fixes suggested there. If the problem persists and you are on a shared system, I would recommend contacting your sysadmin.

Edit: Snakemake docs show that you can override the source cache location by setting the XDG_CACHE_HOME and SNAKEMAKE_OUTPUT_CACHE variables. See here.

Maybe try this before running Snakemake:

export XDG_CACHE_HOME=/path/with/adequate/permissions
export SNAKEMAKE_OUTPUT_CACHE=/path/with/adequate/permissions

Best, Vini

BioInfoStar5 commented 11 months ago

Many thanks Vini, The problem have been solved by allowing more permission (chmod 777) to all snakemake directories. I reentered again this command line ($ metaphor test). However the workflow stop again at an advanced step. See the following log file: log.snakemake.txt Could you see what is the issue here:

Many thanks for your support

vinisalazar commented 11 months ago

Hi @BioInfoStar5,

thank you for reporting that. Your error message shows:

log: output/logs/annotation/cog/download_taxonomy_database.log (check log file(s) for error details)

Could you please paste the contents of that file here?

Best, Vini

BioInfoStar5 commented 11 months ago

Ok, Here is the the content: Config file /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/default-config.yaml is extended by additional config specified via the command line. Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channelpriority strict'. Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/megahit.yaml... Downloading and installing remote packages. Environment for /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/megahit.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/fce7be48d483d8db40988542f82873d4) Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/metabat2.yaml... Downloading and installing remote packages. Environment for /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/metabat2.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/3e6bec9b4505ebdec032a9a2a84bf1c3) Creating conda environment https://github.com/snakemake/snakemake-wrappers/raw/v1.25.0/bio/diamond/makedb/environment.yaml... Downloading and installing remote packages. Environment for https://github.com/snakemake/snakemake-wrappers/raw/v1.25.0/bio/diamond/makedb/environment.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/eedf05d7e57cf3895f126c2d4ed414e8) Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/utils.yaml... Downloading and installing remote packages. Environment for /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/utils.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d) Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/python-utils.yaml... Downloading and installing remote packages. Environment for /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/python-utils.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/8354172c5dce671e23f0abea0df7d840) Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/samtools.yaml... Downloading and installing remote packages. Environment for /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/samtools.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/8cb2960395e92fd996b10b504adf4ffa) Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/prodigal.yaml... Downloading and installing remote packages. Environment for /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/prodigal.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/b1751d0675427529d3db48ba534bed24) Creating conda environment /homeminiconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/vamb.yaml... Downloading and installing remote packages. Environment for /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/vamb.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/2070fd11a55a50241b1e8c3e28fd1837) Creating conda environment https://github.com/snakemake/snakemake-wrappers/raw/v1.25.0/bio/fastp/environment.yaml... Downloading and installing remote packages. Environment for https://github.com/snakemake/snakemake-wrappers/raw/v1.25.0/bio/fastp/environment.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/6eb9aa4dbc6db7efe41919b4e4823b9d) Creating conda environment /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/diamond.yaml... Downloading and installing remote packages. Environment for /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../envs/diamond.yaml created (location: ../../../../home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/dd2a9f97b5a861c9e151e0ce34233381) Creating conda environment https://github.com/snakemake/snakemake-wrappers/raw/v1.25.0/bio/fastqc/environment.yaml... Downloading and installing remote packages. Environment for https://github.com/snakemake/snakemake-wrappers/raw/v1.25.0/bio/fastqc/environment.yaml created (location: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/8254b7fd9265c4f3e4c1e4beb6a8e955) Using shell: /bin/bash Provided cores: 3 Rules claiming more threads will be scaled down. Job stats: job count

all 1 assembly_report 1 cog_functional_parser 1 concatenate_benchmarks 1 concatenate_cog_functional 6 concatenate_contigs 1 concatenate_taxonomies 16 create_contigs_index 1 decompress_catalogue 1 diamond 1 diamond_makedb 1 download_COG_database 1 download_taxonomy_database 1 fastp_pe 4 fastqc_raw 8 fastqc_trimmed 8 generate_COG_taxonmap 1 jgi_summarize_bam_contig_depths 1 lineage_parser 2 map_reads 4 megahit 4 merge_fastqs 8 metabat2 1 plot_benchmarks 1 plot_cog_functional 1 plot_cog_taxonomy 7 prodigal 1 rename_contigs 4 sort_reads 4 taxonomy_parser 1 total 93

Select jobs to execute...

[Thu Aug 3 18:29:42 2023] rule download_taxonomy_database: output: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/rankedlineage.dmp, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/names.dmp, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/nodes.dmp log: output/logs/annotation/cog/download_taxonomy_database.log jobid: 45 reason: Missing output files: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/names.dmp, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/nodes.dmp, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/rankedlineage.dmp resources: tmpdir=/var/tmp

    mkdir -p /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy 2>> output/logs/annotation/cog/download_taxonomy_database.log

    DST=/home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/$(basename https://ftp.ncbi.nih.gov/pub/taxonomy/new_taxdump/new_taxdump.tar.gz)

    # No verbose (nv) option to avoid progress bar
    wget -nv https://ftp.ncbi.nih.gov/pub/taxonomy/new_taxdump/new_taxdump.tar.gz -O $DST 2>> output/logs/annotation/cog/download_taxonomy_database.log

    tar zxvf $DST -C /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy 2>> output/logs/annotation/cog/download_taxonomy_database.log

    ls /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy 2>> output/logs/annotation/cog/download_taxonomy_database.log

Activating conda environment: ../../../../home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d_

[Thu Aug 3 18:29:42 2023] rule fastp_pe: input: /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL2_S001insert_270_short_R1.fq.gz, /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL2_S001insert_270_short_R2.fq.gz output: output/qc/fastp/RL2_S001insert_270_short_single_unit_R1.fq.gz, output/qc/fastp/RL2_S001insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL2_S001insert_270_short_single_unit.html, output/qc/fastp/RL2_S001insert_270_short_single_unit.json log: output/logs/qc/fastp/RL2_S001insert_270_short-single_unit.log jobid: 16 benchmark: output/benchmarks/qc/fastp/RL2_S001insert_270_short-single_unit.txt reason: Missing output files: output/qc/fastp/RL2_S001insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL2_S001insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL2_S001__insert_270_short, unit=single_unit resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

[Thu Aug 3 18:29:42 2023] rule fastp_pe: input: /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL1_S002insert_270_short_R1.fq.gz, /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL1_S002insert_270_short_R2.fq.gz output: output/qc/fastp/RL1_S002insert_270_short_single_unit_R1.fq.gz, output/qc/fastp/RL1_S002insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL1_S002insert_270_short_single_unit.html, output/qc/fastp/RL1_S002insert_270_short_single_unit.json log: output/logs/qc/fastp/RL1_S002insert_270_short-single_unit.log jobid: 13 benchmark: output/benchmarks/qc/fastp/RL1_S002insert_270_short-single_unit.txt reason: Missing output files: output/qc/fastp/RL1_S002insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL1_S002insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL1_S002__insert_270_short, unit=single_unit resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

/home/miniconda2/envs/metaphor/bin/python3.11 /sanhome2/Rachid/DataMicrob/tutorial-metaphor/.snakemake/scripts/tmpbcsb7v5x.wrapper.py Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/6eb9aa4dbc6db7efe41919b4e4823b9d /home/inra/miniconda2/envs/metaphor/bin/python3.11 /sanhome2/Rachid/DataMicrob/tutorial-metaphor/.snakemake/scripts/tmpt0xyrcjp.wrapper.py Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/6eb9aa4dbc6db7efe41919b4e4823b9d [Thu Aug 3 18:29:53 2023] Finished job 13. 1 of 93 steps (1%) done Select jobs to execute...

[Thu Aug 3 18:29:53 2023] rule merge_fastqs: input: output/qc/fastp/RL1_S002insert_270_short_single_unit_R2.fq.gz output: output/qc/merged/RL1_S002insert_270_short_R2.fq.gz log: output/logs/qc/merge_fastqs/RL1_S002insert_270_short.R2.log jobid: 28 benchmark: output/benchmarks/qc/merge_fastqs/RL1_S002insert_270_short.R2.txt reason: Missing output files: output/qc/merged/RL1_S002insert_270_short_R2.fq.gz; Input files updated by another job: output/qc/fastp/RL1_S002insert_270_short_single_unit_R2.fq.gz wildcards: sample=RL1_S002__insert_270_short, read=R2 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL1_S002insert_270_short_single_unit_R2.fq.gz > output/qc/merged/RL1_S002insert_270_short_R2.fq.gz 2> output/logs/qc/merge_fastqs/RL1_S002__insert_270short.R2.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:29:53 2023] Finished job 16. 2 of 93 steps (2%) done Select jobs to execute...

[Thu Aug 3 18:29:53 2023] rule merge_fastqs: input: output/qc/fastp/RL2_S001insert_270_short_single_unit_R2.fq.gz output: output/qc/merged/RL2_S001insert_270_short_R2.fq.gz log: output/logs/qc/merge_fastqs/RL2_S001insert_270_short.R2.log jobid: 32 benchmark: output/benchmarks/qc/merge_fastqs/RL2_S001insert_270_short.R2.txt reason: Missing output files: output/qc/merged/RL2_S001insert_270_short_R2.fq.gz; Input files updated by another job: output/qc/fastp/RL2_S001insert_270_short_single_unit_R2.fq.gz wildcards: sample=RL2_S001__insert_270_short, read=R2 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL2_S001insert_270_short_single_unit_R2.fq.gz > output/qc/merged/RL2_S001insert_270_short_R2.fq.gz 2> output/logs/qc/merge_fastqs/RL2_S001__insert_270short.R2.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:29:53 2023] Finished job 28. 3 of 93 steps (3%) done Select jobs to execute...

[Thu Aug 3 18:29:53 2023] rule merge_fastqs: input: output/qc/fastp/RL1_S002insert_270_short_single_unit_R1.fq.gz output: output/qc/merged/RL1_S002insert_270_short_R1.fq.gz log: output/logs/qc/merge_fastqs/RL1_S002insert_270_short.R1.log jobid: 27 benchmark: output/benchmarks/qc/merge_fastqs/RL1_S002insert_270_short.R1.txt reason: Missing output files: output/qc/merged/RL1_S002insert_270_short_R1.fq.gz; Input files updated by another job: output/qc/fastp/RL1_S002insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL1_S002__insert_270_short, read=R1 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL1_S002insert_270_short_single_unit_R1.fq.gz > output/qc/merged/RL1_S002insert_270_short_R1.fq.gz 2> output/logs/qc/merge_fastqs/RL1_S002__insert_270short.R1.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:29:53 2023] Finished job 32. 4 of 93 steps (4%) done Select jobs to execute...

[Thu Aug 3 18:29:53 2023] rule merge_fastqs: input: output/qc/fastp/RL2_S001insert_270_short_single_unit_R1.fq.gz output: output/qc/merged/RL2_S001insert_270_short_R1.fq.gz log: output/logs/qc/merge_fastqs/RL2_S001insert_270_short.R1.log jobid: 31 benchmark: output/benchmarks/qc/merge_fastqs/RL2_S001insert_270_short.R1.txt reason: Missing output files: output/qc/merged/RL2_S001insert_270_short_R1.fq.gz; Input files updated by another job: output/qc/fastp/RL2_S001insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL2_S001__insert_270_short, read=R1 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL2_S001insert_270_short_single_unit_R1.fq.gz > output/qc/merged/RL2_S001insert_270_short_R1.fq.gz 2> output/logs/qc/merge_fastqs/RL2_S001__insert_270short.R1.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:29:53 2023] Finished job 27. 5 of 93 steps (5%) done Select jobs to execute...

[Thu Aug 3 18:29:53 2023] rule fastp_pe: input: /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL1_S001insert_270_short_R1.fq.gz, /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL1_S001insert_270_short_R2.fq.gz output: output/qc/fastp/RL1_S001insert_270_short_single_unit_R1.fq.gz, output/qc/fastp/RL1_S001insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL1_S001insert_270_short_single_unit.html, output/qc/fastp/RL1_S001insert_270_short_single_unit.json log: output/logs/qc/fastp/RL1_S001insert_270_short-single_unit.log jobid: 10 benchmark: output/benchmarks/qc/fastp/RL1_S001insert_270_short-single_unit.txt reason: Missing output files: output/qc/fastp/RL1_S001insert_270_short_single_unit_R1.fq.gz, output/qc/fastp/RL1_S001insert_270_short_single_unit_R2.fq.gz wildcards: sample=RL1_S001__insert_270_short, unit=single_unit resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

/home/inra/miniconda2/envs/metaphor/bin/python3.11 /sanhome2/Rachid/DataMicrob/tutorial-metaphor/.snakemake/scripts/tmpexzyncsw.wrapper.py Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/6eb9aa4dbc6db7efe41919b4e4823b9d [Thu Aug 3 18:29:53 2023] Finished job 31. 6 of 93 steps (6%) done Select jobs to execute...

[Thu Aug 3 18:29:54 2023] rule download_COG_database: output: /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.fa.gz, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.cog.csv, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.org.csv, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.def.tab, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/fun-20.tab log: output/logs/annotation/cog/download_COG_database.log jobid: 43 benchmark: output/benchmarks/annotation/cog/download_COG_database.txt reason: Missing output files: /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.cog.csv, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.fa.gz, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/fun-20.tab, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.org.csv, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.def.tab resources: tmpdir=/var/tmp

    for file in /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.fa.gz /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.cog.csv /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.org.csv /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.def.tab /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/fun-20.tab; do
        wget https://ftp.ncbi.nih.gov/pub/COG/COG2020/data/$(basename $file) -O $file 2>> output/logs/annotation/cog/download_COG_database.log;
    done

Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d_ [Thu Aug 3 18:30:01 2023] Finished job 10. 7 of 93 steps (8%) done Select jobs to execute...

[Thu Aug 3 18:30:01 2023] rule merge_fastqs: input: output/qc/fastp/RL1_S001insert_270_short_single_unit_R2.fq.gz output: output/qc/merged/RL1_S001insert_270_short_R2.fq.gz log: output/logs/qc/merge_fastqs/RL1_S001insert_270_short.R2.log jobid: 24 benchmark: output/benchmarks/qc/merge_fastqs/RL1_S001insert_270_short.R2.txt reason: Missing output files: output/qc/merged/RL1_S001insert_270_short_R2.fq.gz; Input files updated by another job: output/qc/fastp/RL1_S001insert_270_short_single_unit_R2.fq.gz wildcards: sample=RL1_S001__insert_270_short, read=R2 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL1_S001insert_270_short_single_unit_R2.fq.gz > output/qc/merged/RL1_S001insert_270_short_R2.fq.gz 2> output/logs/qc/merge_fastqs/RL1_S001__insert_270short.R2.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:30:01 2023] Finished job 24. 8 of 93 steps (9%) done Select jobs to execute...

[Thu Aug 3 18:30:02 2023] rule merge_fastqs: input: output/qc/fastp/RL1_S001insert_270_short_single_unit_R1.fq.gz output: output/qc/merged/RL1_S001insert_270_short_R1.fq.gz log: output/logs/qc/merge_fastqs/RL1_S001insert_270_short.R1.log jobid: 23 benchmark: output/benchmarks/qc/merge_fastqs/RL1_S001insert_270_short.R1.txt reason: Missing output files: output/qc/merged/RL1_S001insert_270_short_R1.fq.gz; Input files updated by another job: output/qc/fastp/RL1_S001insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL1_S001__insert_270_short, read=R1 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL1_S001insert_270_short_single_unit_R1.fq.gz > output/qc/merged/RL1_S001insert_270_short_R1.fq.gz 2> output/logs/qc/merge_fastqs/RL1_S001__insert_270short.R1.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:30:02 2023] Finished job 23. 9 of 93 steps (10%) done Select jobs to execute...

[Thu Aug 3 18:30:02 2023] rule fastp_pe: input: /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL2_S002insert_270_short_R1.fq.gz, /sanhome2/Rachid/Data_Microb/tutorial-metaphor/test_data_metaphor/RL2_S002insert_270_short_R2.fq.gz output: output/qc/fastp/RL2_S002insert_270_short_single_unit_R1.fq.gz, output/qc/fastp/RL2_S002insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL2_S002insert_270_short_single_unit.html, output/qc/fastp/RL2_S002insert_270_short_single_unit.json log: output/logs/qc/fastp/RL2_S002insert_270_short-single_unit.log jobid: 19 benchmark: output/benchmarks/qc/fastp/RL2_S002insert_270_short-single_unit.txt reason: Missing output files: output/qc/fastp/RL2_S002insert_270_short_single_unit_R2.fq.gz, output/qc/fastp/RL2_S002insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL2_S002__insert_270_short, unit=single_unit resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

/home/inra/miniconda2/envs/metaphor/bin/python3.11 /sanhome2/Rachid/DataMicrob/tutorial-metaphor/.snakemake/scripts/tmpzmak2ky5.wrapper.py Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/6eb9aa4dbc6db7efe41919b4e4823b9d [Thu Aug 3 18:30:10 2023] Finished job 19. 10 of 93 steps (11%) done Select jobs to execute...

[Thu Aug 3 18:30:10 2023] rule merge_fastqs: input: output/qc/fastp/RL2_S002insert_270_short_single_unit_R2.fq.gz output: output/qc/merged/RL2_S002insert_270_short_R2.fq.gz log: output/logs/qc/merge_fastqs/RL2_S002insert_270_short.R2.log jobid: 36 benchmark: output/benchmarks/qc/merge_fastqs/RL2_S002insert_270_short.R2.txt reason: Missing output files: output/qc/merged/RL2_S002insert_270_short_R2.fq.gz; Input files updated by another job: output/qc/fastp/RL2_S002insert_270_short_single_unit_R2.fq.gz wildcards: sample=RL2_S002__insert_270_short, read=R2 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL2_S002insert_270_short_single_unit_R2.fq.gz > output/qc/merged/RL2_S002insert_270_short_R2.fq.gz 2> output/logs/qc/merge_fastqs/RL2_S002__insert_270short.R2.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:30:10 2023] Finished job 36. 11 of 93 steps (12%) done Select jobs to execute...

[Thu Aug 3 18:30:10 2023] rule merge_fastqs: input: output/qc/fastp/RL2_S002insert_270_short_single_unit_R1.fq.gz output: output/qc/merged/RL2_S002insert_270_short_R1.fq.gz log: output/logs/qc/merge_fastqs/RL2_S002insert_270_short.R1.log jobid: 35 benchmark: output/benchmarks/qc/merge_fastqs/RL2_S002insert_270_short.R1.txt reason: Missing output files: output/qc/merged/RL2_S002insert_270_short_R1.fq.gz; Input files updated by another job: output/qc/fastp/RL2_S002insert_270_short_single_unit_R1.fq.gz wildcards: sample=RL2_S002__insert_270_short, read=R1 resources: tmpdir=/var/tmp, mem_mb=4096, mem_mib=3907

cat output/qc/fastp/RL2_S002insert_270_short_single_unit_R1.fq.gz > output/qc/merged/RL2_S002insert_270_short_R1.fq.gz 2> output/logs/qc/merge_fastqs/RL2_S002__insert_270short.R1.log Activating conda environment: ../../../../home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d [Thu Aug 3 18:30:10 2023] Finished job 35. 12 of 93 steps (13%) done Select jobs to execute... [Thu Aug 3 18:30:47 2023] Error in rule download_COG_database: jobid: 43 output: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.fa.gz, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.cog.csv, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.org.csv, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.def.tab, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/fun-20.tab log: output/logs/annotation/cog/download_COGdatabase.log (check log file(s) for error details) conda-env: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d shell:

    for file in /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.fa.gz /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.cog.csv /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.org.csv /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.def.tab /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/fun-20.tab; do
        wget https://ftp.ncbi.nih.gov/pub/COG/COG2020/data/$(basename $file) -O $file 2>> output/logs/annotation/cog/download_COG_database.log;
    done

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job download_COG_database since they might be corrupted: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/COG2020/cog-20.fa.gz [Thu Aug 3 18:30:51 2023] Error in rule download_taxonomy_database: jobid: 45 output: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/rankedlineage.dmp, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/names.dmp, /home/inra/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/nodes.dmp log: output/logs/annotation/cog/download_taxonomydatabase.log (check log file(s) for error details) conda-env: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d64ee367fa73607c8b22a31cce4e327d shell:

    mkdir -p /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy 2>> output/logs/annotation/cog/download_taxonomy_database.log

    DST=/home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/$(basename https://ftp.ncbi.nih.gov/pub/taxonomy/new_taxdump/new_taxdump.tar.gz)

    # No verbose (nv) option to avoid progress bar
    wget -nv https://ftp.ncbi.nih.gov/pub/taxonomy/new_taxdump/new_taxdump.tar.gz -O $DST 2>> output/logs/annotation/cog/download_taxonomy_database.log

    tar zxvf $DST -C /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy 2>> output/logs/annotation/cog/download_taxonomy_database.log

    ls /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy 2>> output/logs/annotation/cog/download_taxonomy_database.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job download_taxonomy_database since they might be corrupted: /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/rankedlineage.dmp, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/names.dmp, /home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/nodes.dmp Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-08-03T182506.985159.snakemake.log

Thanks

vinisalazar commented 11 months ago

@BioInfoStar5 I believe that is not the correct file. You need the log of the "download_taxonomy_database" rule.

BioInfoStar5 commented 11 months ago

Here is the download_taxonomy_database log file: 2023-08-03 18:30:45 URL:https://ftp.ncbi.nih.gov/pub/taxonomy/new_taxdump/new_taxdump.tar.gz [131291662/131291662] -> "/home/miniconda2/envs/metaphor/lib/python3.11/site-packages/metaphor/config/data/taxonomy/new_taxdump.tar.gz" [1] tar: fullnamelineage.dmp: seulement 8192 octets sur 10240 ont été écrits tar: names.dmp: seulement 5120 octets sur 10240 ont été écrits tar: nodes.dmp : write impossible: No space left on device tar: rankedlineage.dmp : write impossible: No space left on device tar: taxidlineage.dmp : write impossible: No space left on device tar: typematerial.dmp : write impossible: No space left on device tar: typeoftype.dmp : write impossible: No space left on device tar: Arrêt avec code d'échec à cause des erreurs précédentes

I understand that there was no enough space on the directory. isn't it ? Thanks

vinisalazar commented 11 months ago

Hello @BioInfoStar5, yes, that indicates you've run out of disk space while downloading the COG reference database.

Do note that there are multiple ways to install Metaphor in a different volume which has more storage space. The easiest one is probably to create the Metaphor environment in a different location, using the -p (instead of -n) option:

conda create -p /path/with/disk/space -c bioconda metaphor

To activate the environment you'll then need to type:

conda activate /path/with/disk/space

All conda packages and reference data downloaded by Metaphor will go into that environment.

Best, Vini

BioInfoStar5 commented 11 months ago

Thanks a lot Vini ! Regards,

vinisalazar commented 11 months ago

No worries, I'll close this issue for now, but please feel free to reopen it if you continue to have problems.