search

vinisalazar / metaphor

Metaphor: a general-purpose workflow for assembly and binning of metagenomes
https://metaphor-workflow.readthedocs.io/
Other
35 stars 3 forks source link

Error in rule concatenate_contigs #68

Closed francesco-ricci closed 4 months ago

francesco-ricci commented 5 months ago

Hi Vini,

I am having this error, any clue?

Config file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/default-config.yaml is extended by additional config specified via the command line. Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'. Using shell: /bin/bash Provided cores: 48 Rules claiming more threads will be scaled down. Job stats: job count

DAS_tool 1 all 1 assembly_report 1 bins_report 1 cog_functional_parser 1 concatenate_benchmarks 1 concatenate_contigs 1 concatenate_taxonomies 16 concoct 1 create_contigs_index 1 decompress_catalogue 1 diamond 1 fastp_pe 5 index_reads 5 jgi_summarize_bam_contig_depths 1 lineage_parser 2 map_reads 5 megahit 5 merge_fastqs 10 metabat2 1 plot_benchmarks 1 plot_cog_functional 1 prodigal 1 rename_contigs 5 sort_reads 5 taxonomy_parser 1 vamb 1 total 76

Select jobs to execute...

[Wed Jan 17 02:47:18 2024] rule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R2_001.fastq.gz output: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz, output/qc/fastp/SF6228_S311_single_unit.html, output/qc/fastp/SF6228_S311_single_unit.json log: output/logs/qc/fastp/SF6228_S311-single_unit.log jobid: 9 benchmark: output/benchmarks/qc/fastp/SF6228_S311-single_unit.txt reason: Missing output files: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz wildcards: sample=SF6228_S311, unit=single_unit threads: 12 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

[Wed Jan 17 02:47:18 2024] rule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R2_001.fastq.gz output: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz, output/qc/fastp/SF6227_S301_single_unit.html, output/qc/fastp/SF6227_S301_single_unit.json log: output/logs/qc/fastp/SF6227_S301-single_unit.log jobid: 4 benchmark: output/benchmarks/qc/fastp/SF6227_S301-single_unit.txt reason: Missing output files: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz wildcards: sample=SF6227_S301, unit=single_unit threads: 12 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

[Wed Jan 17 02:47:18 2024] rule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz output: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz, output/qc/fastp/SF6229_S321_single_unit.html, output/qc/fastp/SF6229_S321_single_unit.json log: output/logs/qc/fastp/SF6229_S321-single_unit.log jobid: 14 benchmark: output/benchmarks/qc/fastp/SF6229_S321-single_unit.txt reason: Missing output files: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz wildcards: sample=SF6229_S321, unit=single_unit threads: 12 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

[Wed Jan 17 02:47:18 2024] rule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R2_001.fastq.gz output: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz, output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz, output/qc/fastp/SF6231_S341_single_unit.html, output/qc/fastp/SF6231_S341_single_unit.json log: output/logs/qc/fastp/SF6231_S341-single_unit.log jobid: 24 benchmark: output/benchmarks/qc/fastp/SF6231_S341-single_unit.txt reason: Missing output files: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz, output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz wildcards: sample=SF6231_S341, unit=single_unit threads: 12 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_seacorals/.snakemake/scripts/tmprqv2flns.wrapper.py Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_seacorals/.snakemake/scripts/tmpae7zvi8l.wrapper.py Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmpw1tal6ih.wrapper.py /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_seacorals/.snakemake/scripts/tmpwf7uklzc.wrapper.py Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef_ (fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R2_001.fastq.gz --out1 output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz --json output/qc/fastp/SF6227_S301_single_unit.json --html output/qc/fastp/SF6227_S301_single_unit.html ) > output/logs/qc/fastp/SF6227_S301-single_unit.log 2>&1 (fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6231_S34_R2_001.fastq.gz --out1 output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz --json output/qc/fastp/SF6231_S341_single_unit.json --html output/qc/fastp/SF6231_S341_single_unit.html ) > output/logs/qc/fastp/SF6231_S341-single_unit.log 2>&1 (fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R2_001.fastq.gz --out1 output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz --json output/qc/fastp/SF6228_S311_single_unit.json --html output/qc/fastp/SF6228_S311_single_unit.html ) > output/logs/qc/fastp/SF6228_S311-single_unit.log 2>&1 (fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz --out1 output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz --json output/qc/fastp/SF6229_S321_single_unit.json --html output/qc/fastp/SF6229_S321_single_unit.html ) > output/logs/qc/fastp/SF6229_S321-single_unit.log 2>&1 [Wed Jan 17 02:47:30 2024] Finished job 24. 1 of 76 steps (1%) done Select jobs to execute...

[Wed Jan 17 02:47:30 2024] rule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R2_001.fastq.gz output: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz, output/qc/fastp/SF6230_S331_single_unit.html, output/qc/fastp/SF6230_S331_single_unit.json log: output/logs/qc/fastp/SF6230_S331-single_unit.log jobid: 19 benchmark: output/benchmarks/qc/fastp/SF6230_S331-single_unit.txt reason: Missing output files: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz wildcards: sample=SF6230_S331, unit=single_unit threads: 12 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/python3.11 /fs04/rp24/fra/analyses/deep_seacorals/.snakemake/scripts/tmpu9wepsa4.wrapper.py Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef (fastp --thread 12 -M 30 -l 50 --detect_adapter_for_pe --in1 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R1_001.fastq.gz --in2 /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R2_001.fastq.gz --out1 output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz --out2 output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz --json output/qc/fastp/SF6230_S331_single_unit.json --html output/qc/fastp/SF6230_S331_single_unit.html ) > output/logs/qc/fastp/SF6230_S331-single_unit.log 2>&1 [Wed Jan 17 02:49:01 2024] Finished job 9. 2 of 76 steps (3%) done Select jobs to execute...

[Wed Jan 17 02:49:01 2024] rule merge_fastqs: input: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz output: output/qc/merged/SF6228_S311_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6228_S311.R1.log jobid: 8 benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R1.txt reason: Missing output files: output/qc/merged/SF6228_S311_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz wildcards: sample=SF6228_S311, read=R1 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz > output/qc/merged/SF6228_S311_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6228S311.R1.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 02:49:01 2024] rule merge_fastqs: input: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz output: output/qc/merged/SF6231_S341_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6231_S341.R1.log jobid: 23 benchmark: output/benchmarks/qc/merge_fastqs/SF6231_S341.R1.txt reason: Missing output files: output/qc/merged/SF6231_S341_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz wildcards: sample=SF6231_S341, read=R1 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz > output/qc/merged/SF6231_S341_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6231S341.R1.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 02:49:01 2024] rule merge_fastqs: input: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz output: output/qc/merged/SF6228_S311_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6228_S311.R2.log jobid: 10 benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R2.txt reason: Missing output files: output/qc/merged/SF6228_S311_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz wildcards: sample=SF6228_S311, read=R2 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz > output/qc/merged/SF6228_S311_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6228_S311.R2.log

[Wed Jan 17 02:49:01 2024] rule merge_fastqs: input: output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz output: output/qc/merged/SF6231_S341_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6231_S341.R2.log jobid: 25 benchmark: output/benchmarks/qc/merge_fastqs/SF6231_S341.R2.txt reason: Missing output files: output/qc/merged/SF6231_S341_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz wildcards: sample=SF6231_S341, read=R2 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz > output/qc/merged/SF6231_S341_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6231S341.R2.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_ [Wed Jan 17 02:49:02 2024] Finished job 23. 3 of 76 steps (4%) done Removing temporary output output/qc/fastp/SF6231_S341_single_unit_R1.fq.gz. [Wed Jan 17 02:49:02 2024] Finished job 25. 4 of 76 steps (5%) done Removing temporary output output/qc/fastp/SF6231_S341_single_unit_R2.fq.gz. Select jobs to execute... [Wed Jan 17 02:49:03 2024] Finished job 10. 5 of 76 steps (7%) done Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz. [Wed Jan 17 02:49:04 2024] Finished job 8. 6 of 76 steps (8%) done Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz. [Wed Jan 17 02:49:11 2024] Finished job 19. 7 of 76 steps (9%) done

[Wed Jan 17 02:49:11 2024] rule merge_fastqs: input: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz output: output/qc/merged/SF6230_S331_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6230_S331.R2.log jobid: 20 benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R2.txt reason: Missing output files: output/qc/merged/SF6230_S331_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz wildcards: sample=SF6230_S331, read=R2 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz > output/qc/merged/SF6230_S331_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6230S331.R2.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 02:49:11 2024] rule merge_fastqs: input: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz output: output/qc/merged/SF6230_S331_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6230_S331.R1.log jobid: 18 benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R1.txt reason: Missing output files: output/qc/merged/SF6230_S331_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz wildcards: sample=SF6230_S331, read=R1 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz > output/qc/merged/SF6230_S331_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6230S331.R1.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Wed Jan 17 02:49:14 2024] Finished job 20. 8 of 76 steps (11%) done Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz. Select jobs to execute... [Wed Jan 17 02:49:14 2024] Finished job 18. 9 of 76 steps (12%) done Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz. [Wed Jan 17 02:49:16 2024] Finished job 14. 10 of 76 steps (13%) done

[Wed Jan 17 02:49:16 2024] rule merge_fastqs: input: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz output: output/qc/merged/SF6229_S321_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6229_S321.R1.log jobid: 13 benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R1.txt reason: Missing output files: output/qc/merged/SF6229_S321_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz wildcards: sample=SF6229_S321, read=R1 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz > output/qc/merged/SF6229_S321_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6229S321.R1.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 02:49:16 2024] rule merge_fastqs: input: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz output: output/qc/merged/SF6229_S321_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6229_S321.R2.log jobid: 15 benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R2.txt reason: Missing output files: output/qc/merged/SF6229_S321_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz wildcards: sample=SF6229_S321, read=R2 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz > output/qc/merged/SF6229_S321_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6229S321.R2.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Wed Jan 17 02:49:18 2024] Finished job 15. 11 of 76 steps (14%) done Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz. Select jobs to execute... [Wed Jan 17 02:49:18 2024] Finished job 13. 12 of 76 steps (16%) done Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz. [Wed Jan 17 02:49:24 2024] Finished job 4. 13 of 76 steps (17%) done

[Wed Jan 17 02:49:24 2024] rule megahit: input: output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz output: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa log: output/logs/assembly/megahit/SF6230_S331.log jobid: 17 benchmark: output/benchmarks/assembly/megahit/SF6230_S331.txt reason: Missing output files: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa; Input files updated by another job: output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz wildcards: group=SF6230_S331 threads: 48 resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6230_S331

    megahit -1 output/qc/merged/SF6230_S331_R1.fq.gz -2 output/qc/merged/SF6230_S331_R2.fq.gz                       -o output/assembly/megahit/SF6230_S331                       --presets meta-large                                   --out-prefix SF6230_S331                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6230_S331.log

    rm -rf output/assembly/megahit/SF6230_S331/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_ [Wed Jan 17 04:09:59 2024] Finished job 17. 14 of 76 steps (18%) done Select jobs to execute...

[Wed Jan 17 04:10:00 2024] rule megahit: input: output/qc/merged/SF6231_S341_R1.fq.gz, output/qc/merged/SF6231_S341_R2.fq.gz output: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa log: output/logs/assembly/megahit/SF6231_S341.log jobid: 22 benchmark: output/benchmarks/assembly/megahit/SF6231_S341.txt reason: Missing output files: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa; Input files updated by another job: output/qc/merged/SF6231_S341_R2.fq.gz, output/qc/merged/SF6231_S341_R1.fq.gz wildcards: group=SF6231_S341 threads: 48 resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6231_S341

    megahit -1 output/qc/merged/SF6231_S341_R1.fq.gz -2 output/qc/merged/SF6231_S341_R2.fq.gz                       -o output/assembly/megahit/SF6231_S341                       --presets meta-large                                   --out-prefix SF6231_S341                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6231_S341.log

    rm -rf output/assembly/megahit/SF6231_S341/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_ [Wed Jan 17 04:10:17 2024] Finished job 22. 15 of 76 steps (20%) done Select jobs to execute...

[Wed Jan 17 04:10:17 2024] rule megahit: input: output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz output: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa log: output/logs/assembly/megahit/SF6229_S321.log jobid: 12 benchmark: output/benchmarks/assembly/megahit/SF6229_S321.txt reason: Missing output files: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa; Input files updated by another job: output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz wildcards: group=SF6229_S321 threads: 48 resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6229_S321

    megahit -1 output/qc/merged/SF6229_S321_R1.fq.gz -2 output/qc/merged/SF6229_S321_R2.fq.gz                       -o output/assembly/megahit/SF6229_S321                       --presets meta-large                                   --out-prefix SF6229_S321                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6229_S321.log

    rm -rf output/assembly/megahit/SF6229_S321/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_ [Wed Jan 17 05:17:14 2024] Finished job 12. 16 of 76 steps (21%) done Select jobs to execute...

[Wed Jan 17 05:17:14 2024] rule megahit: input: output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6228_S311_R2.fq.gz output: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa log: output/logs/assembly/megahit/SF6228_S311.log jobid: 7 benchmark: output/benchmarks/assembly/megahit/SF6228_S311.txt reason: Missing output files: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa; Input files updated by another job: output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6228_S311_R2.fq.gz wildcards: group=SF6228_S311 threads: 48 resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6228_S311

    megahit -1 output/qc/merged/SF6228_S311_R1.fq.gz -2 output/qc/merged/SF6228_S311_R2.fq.gz                       -o output/assembly/megahit/SF6228_S311                       --presets meta-large                                   --out-prefix SF6228_S311                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6228_S311.log

    rm -rf output/assembly/megahit/SF6228_S311/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_ [Wed Jan 17 06:06:58 2024] Finished job 7. 17 of 76 steps (22%) done Select jobs to execute...

[Wed Jan 17 06:06:58 2024] rule merge_fastqs: input: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz output: output/qc/merged/SF6227_S301_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6227_S301.R2.log jobid: 5 benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R2.txt reason: Missing output files: output/qc/merged/SF6227_S301_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz wildcards: sample=SF6227_S301, read=R2 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz > output/qc/merged/SF6227_S301_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6227S301.R2.log Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 06:06:58 2024] rule rename_contigs: input: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa output: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa log: output/logs/assembly/rename_contigs_SF6228_S311.log jobid: 6 benchmark: output/logs/assembly/rename_contigs_SF6228_S311.txt reason: Missing output files: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa wildcards: group=SF6228_S311 resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa > output/assembly/megahit/SF6228_S311/SF6228_S311.contigsrenamed.fa Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 06:06:58 2024] rule merge_fastqs: input: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz output: output/qc/merged/SF6227_S301_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6227_S301.R1.log jobid: 3 benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R1.txt reason: Missing output files: output/qc/merged/SF6227_S301_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz wildcards: sample=SF6227_S301, read=R1 resources: tmpdir=/tmp, mem_mb=325000, mem_mib=309945

cat output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz > output/qc/merged/SF6227_S301_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6227_S301.R1.log

[Wed Jan 17 06:06:58 2024] Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_ rule rename_contigs: input: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa output: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa log: output/logs/assembly/rename_contigs_SF6230_S331.log jobid: 16 benchmark: output/logs/assembly/rename_contigs_SF6230_S331.txt reason: Missing output files: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa wildcards: group=SF6230_S331 resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa > output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa

[Wed Jan 17 06:06:58 2024] Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb_ rule rename_contigs: input: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa output: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa log: output/logs/assembly/rename_contigs_SF6231_S341.log jobid: 21 benchmark: output/logs/assembly/rename_contigs_SF6231_S341.txt reason: Missing output files: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa wildcards: group=SF6231_S341 resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa > output/assembly/megahit/SF6231_S341/SF6231_S341.contigsrenamed.fa Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Wed Jan 17 06:06:58 2024] rule rename_contigs: input: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa output: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa log: output/logs/assembly/rename_contigs_SF6229_S321.log jobid: 11 benchmark: output/logs/assembly/rename_contigs_SF6229_S321.txt reason: Missing output files: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa wildcards: group=SF6229_S321 resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa > output/assembly/megahit/SF6229_S321/SF6229_S321.contigsrenamed.fa Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Wed Jan 17 06:06:58 2024] Finished job 21. 18 of 76 steps (24%) done Removing temporary output output/assembly/megahit/SF6231_S341/SF6231_S341.contigs.fa. [Wed Jan 17 06:06:59 2024] Finished job 6. 19 of 76 steps (25%) done Removing temporary output output/assembly/megahit/SF6228_S311/SF6228_S311.contigs.fa. [Wed Jan 17 06:06:59 2024] Finished job 11. 20 of 76 steps (26%) done Removing temporary output output/assembly/megahit/SF6229_S321/SF6229_S321.contigs.fa. [Wed Jan 17 06:06:59 2024] Finished job 16. 21 of 76 steps (28%) done Removing temporary output output/assembly/megahit/SF6230_S331/SF6230_S331.contigs.fa. [Wed Jan 17 06:07:00 2024] Finished job 5. 22 of 76 steps (29%) done Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz. [Wed Jan 17 06:07:01 2024] Finished job 3. 23 of 76 steps (30%) done Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz. Select jobs to execute...

[Wed Jan 17 06:07:01 2024] rule megahit: input: output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz output: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa log: output/logs/assembly/megahit/SF6227_S301.log jobid: 2 benchmark: output/benchmarks/assembly/megahit/SF6227_S301.txt reason: Missing output files: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa; Input files updated by another job: output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz wildcards: group=SF6227_S301 threads: 48 resources: tmpdir=/tmp, mem_mb=649968, mem_mib=619858

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/SF6227_S301

    megahit -1 output/qc/merged/SF6227_S301_R1.fq.gz -2 output/qc/merged/SF6227_S301_R2.fq.gz                       -o output/assembly/megahit/SF6227_S301                       --presets meta-large                                   --out-prefix SF6227_S301                              --min-contig-len 1000                    -t 48                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/SF6227_S301.log

    rm -rf output/assembly/megahit/SF6227_S301/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_ [Wed Jan 17 07:57:46 2024] Finished job 2. 24 of 76 steps (32%) done Select jobs to execute...

[Wed Jan 17 07:57:46 2024] rule rename_contigs: input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa output: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa log: output/logs/assembly/rename_contigs_SF6227_S301.log jobid: 1 benchmark: output/logs/assembly/rename_contigs_SF6227_S301.txt reason: Missing output files: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa wildcards: group=SF6227_S301 resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa > output/assembly/megahit/SF6227_S301/SF6227_S301.contigsrenamed.fa Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Wed Jan 17 07:57:50 2024] Finished job 1. 25 of 76 steps (33%) done Removing temporary output output/assembly/megahit/SF6227_S301/SF6227_S301.contigs.fa. Select jobs to execute...

[Wed Jan 17 07:57:50 2024] rule assembly_report: input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa output: output/assembly/assembly_report/avg_length.png, output/assembly/assembly_report/max_length.png, output/assembly/assembly_report/median_length.png, output/assembly/assembly_report/no_bp.png, output/assembly/assembly_report/no_contigs.png, output/assembly/assembly_report/n50.png, output/assembly/assembly_report/assembly_report.tsv log: output/logs/assembly/assembly_report.log jobid: 26 benchmark: output/benchmarks/assembly/assembly_report.txt reason: Missing output files: output/assembly/assembly_report/assembly_report.tsv; Input files updated by another job: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa; Set of input files has changed since last execution; Params have changed since last execution resources: tmpdir=/tmp

[Wed Jan 17 07:57:50 2024] rule concatenate_contigs: input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz log: output/logs/mapping/concatenate_contigs/cobinning.log jobid: 30 benchmark: output/benchmarks/mapping/concatenate_contigs/cobinning.txt reason: Missing output files: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz; Input files updated by another job: output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa; Set of input files has changed since last execution wildcards: group=cobinning resources: tmpdir=/tmp, mem_mb=520000, mem_mib=495911

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log

Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef_ python -c "from future import print_function; import sys, json; print(json.dumps([sys.version_info.major, sys.versioninfo.minor]))" Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a python /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assemblyreport.py Activating conda environment: ../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a /fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assemblyreport.py:89: SyntaxWarning: invalid escape sequence '#' column.replace(" ", "").replace(".", "").replace("#", "n").lower() + ".png" 01/17/2024 07:57:56 Starting script 'assembly_report.py'. 01/17/2024 07:57:56 Full script path: '/fs04/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/scripts/assembly_report.py'.

01/17/2024 07:58:00 could not convert string to float: 'Sequence file is empty.' 01/17/2024 07:58:00 Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/../scripts/utils.py", line 40, in driver main_fn(args) File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 96, in main df = metrics_to_df(args.fastas, args.assembly_report) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 49, in metrics_to_df metrics = [run_seqstats(file) for file in fastas] ^^^^^^^^^^^^^^^^^^ File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 39, in run_seqstats metrics = dict( ^^^^^ File "/fs04/rp24/fra/analyses/deep_sea_corals/.snakemake/scripts/tmp0ngao28t.assembly_report.py", line 40, in (k, float(v)) if "." in v else (k, int(v)) for k, v in metrics.items() ^^^^^^^^ ValueError: could not convert string to float: 'Sequence file is empty.'

Waiting at most 5 seconds for missing files. MissingOutputException in rule assembly_report in file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/assembly.smk, line 103: Job 26 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: output/assembly/assembly_report/avg_length.png output/assembly/assembly_report/max_length.png output/assembly/assembly_report/median_length.png output/assembly/assembly_report/no_bp.png output/assembly/assembly_report/no_contigs.png output/assembly/assembly_report/n50.png output/assembly/assembly_report/assembly_report.tsv [Wed Jan 17 07:58:11 2024] Error in rule concatenate_contigs: jobid: 30 input: output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa, output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa, output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa, output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa, output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz log: output/logs/mapping/concatenatecontigs/cobinning.log (check log file(s) for error details) conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef shell:

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/SF6227_S301/SF6227_S301.contigs_renamed.fa output/assembly/megahit/SF6228_S311/SF6228_S311.contigs_renamed.fa output/assembly/megahit/SF6229_S321/SF6229_S321.contigs_renamed.fa output/assembly/megahit/SF6230_S331/SF6230_S331.contigs_renamed.fa output/assembly/megahit/SF6231_S341/SF6231_S341.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job concatenate_contigs since they might be corrupted: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-01-17T024534.473506.snakemake.log Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/metaphor", line 10, in sys.exit(main()) ^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 278, in main args.func(args) File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/execute.py", line 109, in main retcode = run_cmd(cmd) ^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd retcode = check_call(cmd.split()) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', 'metaphor_settings.yaml', '--cores', '48', '-p', '-r', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda', '--config', 'samples=samples.csv', '--config', 'max_mb=650000']' returned non-zero exit status 1.

vinisalazar commented 5 months ago

Hi @francesco-ricci,

Thank you for reporting this.

What version of Metaphor, Snakemake and Python are you using? There have been a few updates recently with the release of Snakemake 8 and it's better to make sure everything is up-to-date.

Could you also print the contents of this log file: output/logs/mapping/concatenate_contigs/cobinning.log

Thanks Vini

francesco-ricci commented 5 months ago

Hey Vini,

snakemae 7.32.4, Python 3.9.18, metaphor v1.7.9.

Print:

Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c322$ rename=(not args.keepnames)) File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c322$ for entry in entries: File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c322$ raise ValueError('Empty or outcommented file') ValueError: Empty or outcommented file

vinisalazar commented 5 months ago

I think those versions are stable but I recommend creating a new environment from scratch, with Metaphor and Snakemake. Your versions should be: Metaphor=1.7.10, Snakemake>=8, Python=3.11.*

From that and your other log, it seems some of your assembly files were created but some may be empty. Have a look, if that's the case, delete the empty ones and resume the analysis, it should complete the missing assemblies and continue.

Let me know how it goes, Vini

francesco-ricci commented 5 months ago

Thanks Vini!

francesco-ricci commented 5 months ago

Hey Vini,

so I installed all the versions you said but now I have a problem with checkm, cause it's not compatible with python3.11. Any advice?

francesco-ricci commented 5 months ago

This is the error:

(metaphor) [fricci@m3-login2 deep_sea_corals]$ mamba install bioconda::checkm-genome python=3.11

Looking for: ['bioconda::checkm-genome', 'python=3.11']

bioconda/linux-64 Using cache bioconda/noarch Using cache conda-forge/linux-64 Using cache conda-forge/noarch Using cache bioconda/linux-64 Using cache bioconda/noarch Using cache pkgs/r/noarch No change pkgs/main/linux-64 No change pkgs/main/noarch No change pkgs/r/linux-64 No change Could not solve for environment specs The following packages are incompatible ├─ checkm-genome is installable with the potential options │ ├─ checkm-genome [1.0.11|1.0.12|...|1.0.7] would require │ │ └─ python [2.7 |2.7. |>=2.7,<2.8.0a0 ], which can be installed; │ ├─ checkm-genome 1.0.11 would require │ │ └─ python <3 , which can be installed; │ ├─ checkm-genome 1.0.7 would require │ │ └─ python 3.5 , which can be installed; │ ├─ checkm-genome 1.0.7 would require │ │ └─ python 3.6 , which can be installed; │ ├─ checkm-genome [1.1.0|1.1.1|1.1.2|1.1.3] would require │ │ └─ pysam >=0.12.0.1 with the potential options │ │ ├─ pysam [0.12.0.1|0.13.0|...|0.15.4] would require │ │ │ └─ python [2.7 |>=2.7,<2.8.0a0 ], which can be installed; │ │ ├─ pysam [0.12.0.1|0.13.0|...|0.15.2] would require │ │ │ └─ python [3.5 |>=3.5,<3.6.0a0 ], which can be installed; │ │ ├─ pysam [0.12.0.1|0.13.0|0.14.0|0.14.1] would require │ │ │ └─ python 3.6 , which can be installed; │ │ ├─ pysam [0.14.1|0.15.0|...|0.15.4] would require │ │ │ └─ python >=3.6,<3.7.0a0 , which can be installed; │ │ ├─ pysam [0.15.2|0.16.0.1|...|0.20.0] would require │ │ │ ├─ python >=2.7,<2.8.0a0 , which can be installed; │ │ │ └─ python_abi 2.7. _cp27mu, which can be installed; │ │ ├─ pysam [0.15.2|0.15.4|...|0.21.0] would require │ │ │ ├─ python >=3.6,<3.7.0a0 , which can be installed; │ │ │ └─ python_abi 3.6. _cp36m, which can be installed; │ │ ├─ pysam [0.15.2|0.15.4|...|0.21.0] would require │ │ │ ├─ python >=3.7,<3.8.0a0 , which can be installed; │ │ │ └─ python_abi 3.7. _cp37m, which can be installed; │ │ ├─ pysam [0.15.2|0.15.3|0.15.4] would require │ │ │ └─ python >=3.7,<3.8.0a0 , which can be installed; │ │ ├─ pysam [0.15.2|0.16.0.1|...|0.22.0] would require │ │ │ ├─ python >=3.8,<3.9.0a0 , which can be installed; │ │ │ └─ python_abi 3.8. _cp38, which can be installed; │ │ ├─ pysam [0.16.0.1|0.17.0|...|0.22.0] would require │ │ │ ├─ python >=3.9,<3.10.0a0 , which can be installed; │ │ │ └─ python_abi 3.9. _cp39, which can be installed; │ │ └─ pysam [0.19.1|0.20.0|0.21.0|0.22.0] would require │ │ ├─ python >=3.10,<3.11.0a0 , which can be installed; │ │ └─ python_abi 3.10. *_cp310, which can be installed; │ └─ checkm-genome [1.1.10|1.1.11|...|1.2.2] would require │ └─ pysam >=0.15.3 , which can be installed (as previously explained); └─ python 3.11* is not installable because there are no viable options ├─ python [3.11.0|3.11.2|...|3.11.7] conflicts with any installable versions previously reported; └─ python [3.11.0|3.11.1|...|3.11.7] would require └─ python_abi 3.11. *_cp311, which conflicts with any installable versions previously reported.

vinisalazar commented 5 months ago

Metaphor does not use CheckM, maybe install it in a separate environment?

francesco-ricci commented 5 months ago

Damn right!

Anyway, I am having a similar issue again, can you help me solve it Vini?

Config file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/default-config.yaml is extended by additional config specified via the command line. Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'. Retrieving input from storage. Using shell: /bin/bash Provided cores: 64 Rules claiming more threads will be scaled down. Job stats: job count

DAS_tool 1 all 1 assembly_report 1 bins_report 1 cog_functional_parser 1 concatenate_benchmarks 1 concatenate_contigs 1 concatenate_taxonomies 16 concoct 1 create_contigs_index 1 decompress_catalogue 1 diamond 1 fastp_pe 4 index_reads 4 jgi_summarize_bam_contig_depths 1 lineage_parser 2 map_reads 4 megahit 1 merge_fastqs 8 metabat2 1 plot_benchmarks 1 plot_cog_functional 1 prodigal 1 rename_contigs 1 sort_reads 4 taxonomy_parser 1 vamb 1 total 62

Select jobs to execute... Execute 4 jobs...

[Thu Jan 25 16:12:43 2024] localrule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6230_S33_R2_001.fastq.gz output: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz, output/qc/fastp/SF6230_S331_single_unit.html, output/qc/fastp/SF6230_S331_single_unit.json log: output/logs/qc/fastp/SF6230_S331-single_unit.log jobid: 10 benchmark: output/benchmarks/qc/fastp/SF6230_S331-single_unit.txt reason: Missing output files: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz, output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz wildcards: sample=SF6230_S331, unit=single_unit threads: 16 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

[Thu Jan 25 16:12:43 2024] localrule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R2_001.fastq.gz output: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz, output/qc/fastp/SF6227_S301_single_unit.html, output/qc/fastp/SF6227_S301_single_unit.json log: output/logs/qc/fastp/SF6227_S301-single_unit.log jobid: 4 benchmark: output/benchmarks/qc/fastp/SF6227_S301-single_unit.txt reason: Missing output files: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz, output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz wildcards: sample=SF6227_S301, unit=single_unit threads: 16 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

[Thu Jan 25 16:12:43 2024] localrule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6228_S31_R2_001.fastq.gz output: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz, output/qc/fastp/SF6228_S311_single_unit.html, output/qc/fastp/SF6228_S311_single_unit.json log: output/logs/qc/fastp/SF6228_S311-single_unit.log jobid: 6 benchmark: output/benchmarks/qc/fastp/SF6228_S311-single_unit.txt reason: Missing output files: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz, output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz wildcards: sample=SF6228_S311, unit=single_unit threads: 16 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

[Thu Jan 25 16:12:44 2024] localrule fastp_pe: input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R1_001.fastq.gz, /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz output: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz, output/qc/fastp/SF6229_S321_single_unit.html, output/qc/fastp/SF6229_S321_single_unit.json log: output/logs/qc/fastp/SF6229_S321-single_unit.log jobid: 8 benchmark: output/benchmarks/qc/fastp/SF6229_S321-single_unit.txt reason: Missing output files: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz, output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz wildcards: sample=SF6229_S321, unit=single_unit threads: 16 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/326d14fb55a844699add8570e17ae5ef [Thu Jan 25 16:13:52 2024] Finished job 6. 1 of 62 steps (2%) done Select jobs to execute... Execute 2 jobs...

[Thu Jan 25 16:13:52 2024] localrule merge_fastqs: input: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz output: output/qc/merged/SF6228_S311_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6228_S311.R2.log jobid: 12 benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R2.txt reason: Missing output files: output/qc/merged/SF6228_S311_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz wildcards: sample=SF6228_S311, read=R2 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz > output/qc/merged/SF6228_S311_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6228S311.R2.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Thu Jan 25 16:13:54 2024] localrule merge_fastqs: input: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz output: output/qc/merged/SF6228_S311_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6228_S311.R1.log jobid: 5 benchmark: output/benchmarks/qc/merge_fastqs/SF6228_S311.R1.txt reason: Missing output files: output/qc/merged/SF6228_S311_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz wildcards: sample=SF6228_S311, read=R1 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz > output/qc/merged/SF6228_S311_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6228S311.R1.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Thu Jan 25 16:13:56 2024] Finished job 12. 2 of 62 steps (3%) done Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R2.fq.gz. [Thu Jan 25 16:13:57 2024] Finished job 5. 3 of 62 steps (5%) done Removing temporary output output/qc/fastp/SF6228_S311_single_unit_R1.fq.gz. [Thu Jan 25 16:13:57 2024] Finished job 10. 4 of 62 steps (6%) done Select jobs to execute... Execute 2 jobs...

[Thu Jan 25 16:13:57 2024] localrule merge_fastqs: input: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz output: output/qc/merged/SF6230_S331_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6230_S331.R2.log jobid: 14 benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R2.txt reason: Missing output files: output/qc/merged/SF6230_S331_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz wildcards: sample=SF6230_S331, read=R2 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz > output/qc/merged/SF6230_S331_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6230S331.R2.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Thu Jan 25 16:13:57 2024] localrule merge_fastqs: input: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz output: output/qc/merged/SF6230_S331_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6230_S331.R1.log jobid: 9 benchmark: output/benchmarks/qc/merge_fastqs/SF6230_S331.R1.txt reason: Missing output files: output/qc/merged/SF6230_S331_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz wildcards: sample=SF6230_S331, read=R1 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz > output/qc/merged/SF6230_S331_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6230S331.R1.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Thu Jan 25 16:13:59 2024] Finished job 8. 5 of 62 steps (8%) done Select jobs to execute... Execute 2 jobs...

[Thu Jan 25 16:13:59 2024] localrule merge_fastqs: input: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz output: output/qc/merged/SF6229_S321_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6229_S321.R2.log jobid: 13 benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R2.txt reason: Missing output files: output/qc/merged/SF6229_S321_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz wildcards: sample=SF6229_S321, read=R2 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz > output/qc/merged/SF6229_S321_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6229S321.R2.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Thu Jan 25 16:14:00 2024] localrule merge_fastqs: input: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz output: output/qc/merged/SF6229_S321_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6229_S321.R1.log jobid: 7 benchmark: output/benchmarks/qc/merge_fastqs/SF6229_S321.R1.txt reason: Missing output files: output/qc/merged/SF6229_S321_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz wildcards: sample=SF6229_S321, read=R1 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz > output/qc/merged/SF6229_S321_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6229S321.R1.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Thu Jan 25 16:14:00 2024] Finished job 14. 6 of 62 steps (10%) done Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R2.fq.gz. [Thu Jan 25 16:14:01 2024] Finished job 9. 7 of 62 steps (11%) done Removing temporary output output/qc/fastp/SF6230_S331_single_unit_R1.fq.gz. [Thu Jan 25 16:14:08 2024] Finished job 7. 8 of 62 steps (13%) done Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R1.fq.gz. [Thu Jan 25 16:14:08 2024] Finished job 4. 9 of 62 steps (15%) done Select jobs to execute... Execute 2 jobs...

[Thu Jan 25 16:14:08 2024] localrule merge_fastqs: input: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz output: output/qc/merged/SF6227_S301_R2.fq.gz log: output/logs/qc/merge_fastqs/SF6227_S301.R2.log jobid: 11 benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R2.txt reason: Missing output files: output/qc/merged/SF6227_S301_R2.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz wildcards: sample=SF6227_S301, read=R2 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz > output/qc/merged/SF6227_S301_R2.fq.gz 2> output/logs/qc/merge_fastqs/SF6227S301.R2.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb

[Thu Jan 25 16:14:08 2024] localrule merge_fastqs: input: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz output: output/qc/merged/SF6227_S301_R1.fq.gz log: output/logs/qc/merge_fastqs/SF6227_S301.R1.log jobid: 3 benchmark: output/benchmarks/qc/merge_fastqs/SF6227_S301.R1.txt reason: Missing output files: output/qc/merged/SF6227_S301_R1.fq.gz; Input files updated by another job: output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz wildcards: sample=SF6227_S301, read=R1 resources: tmpdir=/tmp, mem_mb=500000, mem_mib=476838

cat output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz > output/qc/merged/SF6227_S301_R1.fq.gz 2> output/logs/qc/merge_fastqs/SF6227S301.R1.log Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Thu Jan 25 16:14:08 2024] Finished job 13. 10 of 62 steps (16%) done Removing temporary output output/qc/fastp/SF6229_S321_single_unit_R2.fq.gz. [Thu Jan 25 16:14:13 2024] Finished job 11. 11 of 62 steps (18%) done Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R2.fq.gz. [Thu Jan 25 16:14:13 2024] Finished job 3. 12 of 62 steps (19%) done Removing temporary output output/qc/fastp/SF6227_S301_single_unit_R1.fq.gz. Select jobs to execute... Execute 1 jobs...

[Thu Jan 25 16:14:13 2024] localrule megahit: input: output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz, output/qc/merged/SF6228_S311_R2.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz output: output/assembly/megahit/coassembly/coassembly.contigs.fa log: output/logs/assembly/megahit/coassembly.log jobid: 2 benchmark: output/benchmarks/assembly/megahit/coassembly.txt reason: Missing output files: output/assembly/megahit/coassembly/coassembly.contigs.fa; Input files updated by another job: output/qc/merged/SF6228_S311_R2.fq.gz, output/qc/merged/SF6227_S301_R2.fq.gz, output/qc/merged/SF6227_S301_R1.fq.gz, output/qc/merged/SF6229_S321_R1.fq.gz, output/qc/merged/SF6230_S331_R2.fq.gz, output/qc/merged/SF6228_S311_R1.fq.gz, output/qc/merged/SF6230_S331_R1.fq.gz, output/qc/merged/SF6229_S321_R2.fq.gz wildcards: group=coassembly threads: 64 resources: tmpdir=/tmp, mem_mb=1000000, mem_mib=953675

    # MegaHit has no --force flag, so we must remove the created directory prior to running
    rm -rf output/assembly/megahit/coassembly

    megahit -1 output/qc/merged/SF6227_S301_R1.fq.gz,output/qc/merged/SF6228_S311_R1.fq.gz,output/qc/merged/SF6229_S321_R1.fq.gz,output/qc/merged/SF6230_S331_R1.fq.gz -2 output/qc/merged/SF6227_S301_R2.fq.gz,output/qc/merged/SF6228_S311_R2.fq.gz,output/qc/merged/SF6229_S321_R2.fq.gz,output/qc/merged/SF6230_S331_R2.fq.gz                       -o output/assembly/megahit/coassembly                       --presets meta-large                                   --out-prefix coassembly                              --min-contig-len 1000                    -t 64                                                --k-list 21,29,39,59,79,99,119,141 &> output/logs/assembly/megahit/coassembly.log

    rm -rf output/assembly/megahit/coassembly/intermediate_contigs  # to avoid this, disable the 'remove_intermediate_contigs' setting in the config file.

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/f98ddf6c1cb52f3d784c38b4fd62b9fd_ [Thu Jan 25 18:55:47 2024] Finished job 2. 13 of 62 steps (21%) done Select jobs to execute... Execute 1 jobs...

[Thu Jan 25 18:55:47 2024] localrule rename_contigs: input: output/assembly/megahit/coassembly/coassembly.contigs.fa output: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa log: output/logs/assembly/rename_contigs_coassembly.log jobid: 1 benchmark: output/logs/assembly/rename_contigs_coassembly.txt reason: Missing output files: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa; Input files updated by another job: output/assembly/megahit/coassembly/coassembly.contigs.fa wildcards: group=coassembly resources: tmpdir=/tmp

awk '/^>/{gsub(" |\\.|=", "_", $0); print $0; next}{print}' output/assembly/megahit/coassembly/coassembly.contigs.fa > output/assembly/megahit/coassembly/coassembly.contigsrenamed.fa Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/d3358a3e08c706bbaa9bdef9878cc2fb [Thu Jan 25 18:55:49 2024] Finished job 1. 14 of 62 steps (23%) done Removing temporary output output/assembly/megahit/coassembly/coassembly.contigs.fa. Select jobs to execute... Execute 2 jobs...

[Thu Jan 25 18:55:49 2024] localrule concatenate_contigs: input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz log: output/logs/mapping/concatenate_contigs/cobinning.log jobid: 19 benchmark: output/benchmarks/mapping/concatenate_contigs/cobinning.txt reason: Missing output files: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz; Input files updated by another job: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa wildcards: group=cobinning resources: tmpdir=/tmp, mem_mb=800000, mem_mib=762940

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef_

[Thu Jan 25 18:55:49 2024] localrule assembly_report: input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa output: output/assembly/assembly_report/avg_length.png, output/assembly/assembly_report/max_length.png, output/assembly/assembly_report/median_length.png, output/assembly/assembly_report/no_bp.png, output/assembly/assembly_report/no_contigs.png, output/assembly/assembly_report/n50.png, output/assembly/assembly_report/assembly_report.tsv log: output/logs/assembly/assembly_report.log jobid: 15 benchmark: output/benchmarks/assembly/assembly_report.txt reason: Missing output files: output/assembly/assembly_report/assembly_report.tsv; Input files updated by another job: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa resources: tmpdir=/tmp

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_ [Thu Jan 25 18:55:49 2024] Error in rule concatenate_contigs: jobid: 19 input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa output: output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz log: output/logs/mapping/concatenatecontigs/cobinning.log (check log file(s) for error details) conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/62c4f57b116324c01c3221a9b5c2a0ef shell:

    concatenate.py -m 1000 output/mapping/cobinning/cobinning_contigs_catalogue.fna.gz output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa &> output/logs/mapping/concatenate_contigs/cobinning.log

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Activating conda environment: ../../../../../../home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a_ Traceback (most recent call last): File "/fs04/rp24/fra/analyses/deep_sea_corals/genome_resolved_analysis_coassembly_cobinning/.snakemake/scripts/tmp5p_p4byu.assembly_report.py", line 11, in import pandas as pd File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/pandas/init.py", line 46, in from pandas.core.api import ( File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/pandas/core/api.py", line 1, in from pandas._libs import ( File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/pandas/_libs/init.py", line 16, in import pandas._libs.pandas_parser # noqa: E501 # isort: skip # type: ignore[reportUnusedImport] ModuleNotFoundError: No module named 'pandas._libs.pandasparser' RuleException: CalledProcessError in file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/assembly.smk, line 134: Command 'source /home/fricci/rp24/fra/software/miniconda3/bin/activate '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a'; set -euo pipefail; python /fs04/rp24/fra/analyses/deep_sea_corals/genome_resolved_analysis_coassembly_cobinning/.snakemake/scripts/tmp5p_p4byu.assembly_report.py' returned non-zero exit status 1. [Thu Jan 25 18:55:51 2024] Error in rule assembly_report: jobid: 15 input: output/assembly/megahit/coassembly/coassembly.contigs_renamed.fa output: output/assembly/assembly_report/avg_length.png, output/assembly/assembly_report/max_length.png, output/assembly/assembly_report/median_length.png, output/assembly/assembly_report/no_bp.png, output/assembly/assembly_report/no_contigs.png, output/assembly/assembly_report/n50.png, output/assembly/assembly_report/assembly_report.tsv log: output/logs/assembly/assemblyreport.log (check log file(s) for error details) conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/7fa2c3b43c2607597842de1611da163a

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-01-25T161219.079538.snakemake.log WorkflowError: At least one job did not complete successfully. Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/metaphor", line 10, in sys.exit(main()) ^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 278, in main args.func(args) File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/execute.py", line 109, in main retcode = run_cmd(cmd) ^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd retcode = check_call(cmd.split()) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', 'metaphor_settings.yaml', '--cores', '64', '--printshellcmds', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda', '--config', 'samples=samples.csv', '--config', 'max_mb=1000000']' returned non-zero exit status 1.

vinisalazar commented 5 months ago

Looks like an issue with the version of Pandas. I can submit a fix by Monday.

francesco-ricci commented 5 months ago

Thanks Vini!On 26 Jan 2024, at 16:46, Vini Salazar @.***> wrote: Looks like an issue with the version of Pandas. I can submit a fix by Monday.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

vinisalazar commented 5 months ago

@francesco-ricci could you please confirm the Pandas and Python version in your metaphor conda environment?

Thank you

francesco-ricci commented 5 months ago

Pandas 2.1.4 (I just updated it to 2.2.0); python 3.10.13

Thanks Francesco

Dr Francesco Ricci Postdoctoral Research Fellow, Greening Lab https://streaklinks.com/B1QGa0IR0WtvIhzTbApbDp80/http%3A%2F%2Fwww.greeninglab.com%2F Department of Microbiology Monash University - Address: 19 Innovation Walk, Office 135, Clayton VIC 3800 Australia

ᐧ

On Mon, 29 Jan 2024 at 14:20, Vini Salazar @.***> wrote:

@francesco-ricci https://github.com/francesco-ricci could you please confirm the Pandas and Python version in your metaphor conda environment?

Thank you

— Reply to this email directly, view it on GitHub https://github.com/vinisalazar/metaphor/issues/68#issuecomment-1913893649, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANALJT2D7QB33SK7JIICDFDYQ4IOTAVCNFSM6AAAAABB5SH5YKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTHA4TGNRUHE . You are receiving this because you were mentioned.Message ID: @.***>

vinisalazar commented 5 months ago

Hi @francesco-ricci,

I looked into this one but couldn't find any obvious solutions.

Could you please try:

The last step should force the recreation of the conda environment used in the offending rule (assembly_report)

I see that the import error that is happening: ModuleNotFoundError: No module named 'pandas.libs.pandas_parser' is expected in Pandas<=2.0.*, but if the version is 2.1.* or above, I wouldn't expect it to happen.

Let me know how it goes and if it doesn't work, I can submit a patch with an option to turn off the assembly report.

Thanks, Vini

francesco-ricci commented 5 months ago

Hi VIni,

this worked! thank you!