search

marbl / MetaCompass

MetaCompass: Reference-guided Assembly of Metagenomes
https://github.com/marbl/MetaCompass/wiki
Other
38 stars 11 forks source link

TypeError in line 133 of metacompass.iter0.ref.py, returned non-zero exit status 1 if TypeError resolved #7

Closed alexdthomas closed 5 years ago

alexdthomas commented 5 years ago

Hello,

I'm trying to run the tutorial example with the new release (https://github.com/marbl/MetaCompass/releases/tag/paper-v1.0). After checking dependencies and installing I ran python go_metacompass.py -r tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta -P tutorial/thao2000.1.fq,tutorial/thao2000.2.fq -o example1_output -m 1 -t 4 the output is 

confirming file containing reference genomes exists..
[OK]
checking for dependencies (Bowtie2, Blast, kmermask, Snakemake)
Bowtie2--->[OK]
/usr/bin/blastn
Blast+--->[OK]
/home/talex/apps/MetaCompass-paper-v1.0/bin/kmer-mask
kmer-mask--->[OK]
/home/talex/.pyenv/versions/miniconda3-latest/bin/snakemake
Snakemake--->[OK]
Full Traceback (most recent call last):
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/__init__.py", line 296, in snakemake
    print_compilation=print_compilation)
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/workflow.py", line 522, in include
    exec(compile(code, snakefile, "exec"), self.globals)
  File "/home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py", line 314, in <module>
TypeError: %d format: a number is required, not str

TypeError in line 133 of /home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py:
%d format: a number is required, not str
  File "/home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py", line 133, in <module>
ERROR: snakemake command failed; exiting..
touch: cannot touch 'example1_output/thao2000.0.assembly.out/run.fail': No such file or directory

I tried modifying line 133 of MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py as either 1) changing %d to %s shell:"rm -rf %s/%s.0.assembly.out/%s.megahit; megahit -o %s/%s.0.assembly.out/%s.megahit --min-count %s --min-contig-len %s --presets meta-sensitive -t {threads} -1 {input.r1} -2 {input.r2} 1>> {log} 2>&1"%(config['prefix'],config['sample'],config['sample'],config['prefix'],config['sample'],config['sample'],config['mincov'],config['minlen']) 2) hardcoding values shell:"rm -rf %s/%s.0.assembly.out/%s.megahit; megahit -o %s/%s.0.assembly.out/%s.megahit --min-count 3 --min-contig-len 300 --presets meta-sensitive -t {threads} -1 {input.r1} -2 {input.r2} 1>> {log} 2>&1"%(config['prefix'],config['sample'],config['sample'],config['prefix'],config['sample'],config['sample'])

both modifications resolve the immediate TypeError, but produce the following error instead... 

confirming file containing reference genomes exists..                                                                                                                                                [100/46216]
[OK]
checking for dependencies (Bowtie2, Blast, kmermask, Snakemake)
Bowtie2--->[OK]
/usr/bin/blastn
Blast+--->[OK]
/home/talex/apps/MetaCompass-paper-v1.0/bin/kmer-mask
kmer-mask--->[OK]
/home/talex/.pyenv/versions/miniconda3-latest/bin/snakemake
Snakemake--->[OK]
Provided cores: 4
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       assemble_unmapped
        1       bam_sort
        1       bowtie2_map
        1       build_contigs
        1       join_contigs
        1       merge_reads
        1       pilon_contigs
        1       pilon_map
        1       sam_to_bam
        9
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
        merge_reads
Selected jobs (1):
        merge_reads
Resources after job selection: {'_cores': 3, '_nodes': 9223372036854775806}
---merge fastq reads
Releasing 1 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
1 of 9 steps (11%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
        bowtie2_map
Selected jobs (1):
        bowtie2_map
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}
---Build index .
Releasing 4 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
2 of 9 steps (22%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
        build_contigs
Selected jobs (1):
        build_contigs
Resources after job selection: {'_cores': 3, '_nodes': 9223372036854775806}
---Build contigs .
Releasing 1 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
3 of 9 steps (33%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
        pilon_map
Selected jobs (1):
        pilon_map
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}
---Map reads for pilon polishing.
Releasing 4 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
4 of 9 steps (44%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (2):
        assemble_unmapped
        sam_to_bam
Selected jobs (1):
        sam_to_bam
Resources after job selection: {'_cores': 3, '_nodes': 9223372036854775806}
---Convert sam to bam .
Releasing 1 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
5 of 9 steps (56%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (2):
        bam_sort
        assemble_unmapped
Selected jobs (1):
        bam_sort
Resources after job selection: {'_cores': 3, '_nodes': 9223372036854775806}
---Sort bam .
Releasing 1 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
6 of 9 steps (67%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (2):
        assemble_unmapped
        pilon_contigs
Selected jobs (1):
        pilon_contigs
Resources after job selection: {'_cores': 3, '_nodes': 9223372036854775806}
---Pilon polish contigs .
Releasing 1 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
7 of 9 steps (78%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
        assemble_unmapped
Selected jobs (1):
        assemble_unmapped
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}
---Assemble unmapped reads .
Full Traceback (most recent call last):
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/executors.py", line 784, in run_wrapper
    version)
  File "/home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py", line 349, in __rule_assemble_unmapped
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/shell.py", line 74, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'rm -rf example1_output/thao2000.0.assembly.out/thao2000.megahit; megahit -o example1_output/thao2000.0.assembly.out/thao2000.megahit --min-count 3 --min-contig-len 300 --presets meta-sensitive -t 4 -1 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.1.fq -2 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.2.fq  1>> example1_output/thao2000.0.megahit.log 2>&1' returned non-zero exit status 1

Error in job assemble_unmapped while creating output file example1_output/thao2000.0.assembly.out/thao2000.megahit/final.contigs.fa.
Full Traceback (most recent call last):
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/executors.py", line 784, in run_wrapper
    version)
  File "/home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py", line 349, in __rule_assemble_unmapped
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/shell.py", line 74, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'rm -rf example1_output/thao2000.0.assembly.out/thao2000.megahit; megahit -o example1_output/thao2000.0.assembly.out/thao2000.megahit --min-count 3 --min-contig-len 300 --presets meta-sensitive -t 4 -1 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.1.fq -2 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.2.fq  1>> example1_output/thao2000.0.megahit.log 2>&1' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/executors.py", line 247, in _callback
    raise ex
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/site-packages/snakemake/executors.py", line 798, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 133 of /home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py:
Command 'rm -rf example1_output/thao2000.0.assembly.out/thao2000.megahit; megahit -o example1_output/thao2000.0.assembly.out/thao2000.megahit --min-count 3 --min-contig-len 300 --presets meta-sensitive -t 4 -1 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.1.fq -2 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.2.fq  1>> example1_output/thao2000.0.megahit.log 2>&1' returned non-zero exit status 1
  File "/home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py", line 133, in __rule_assemble_unmapped

RuleException:
CalledProcessError in line 133 of /home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py:
Command 'rm -rf example1_output/thao2000.0.assembly.out/thao2000.megahit; megahit -o example1_output/thao2000.0.assembly.out/thao2000.megahit --min-count 3 --min-contig-len 300 --presets meta-sensitive -t 4 -1 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.1.fq -2 example1_output/thao2000.0.assembly.out/thao2000.mc.sam.unmapped.2.fq  1>> example1_output/thao2000.0.megahit.log 2>&1' returned non-zero exit status 1
  File "/home/talex/apps/MetaCompass-paper-v1.0/snakemake/metacompass.iter0.ref.py", line 133, in __rule_assemble_unmapped
  File "/home/talex/.pyenv/versions/miniconda3-latest/lib/python3.5/concurrent/futures/thread.py", line 55, in run
Releasing 4 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message
unlocking
removing lock
removing lock
removed all locks
ERROR: snakemake command failed; exiting..

I thought maybe it was just an error to the megahit call (using version 1.1.3), but the only possibly issue I see is that --presets meta-sensitive overrides --min-count (removing the --min-count arg from line 133 has no effect though).

My only guess might be an issue with snakemake (which I have no experience with) or the object class in the initial construction of the config object (which throws the original TypeError). At least the previous issue I raised (#5) seems to be resolved. I am looking forward to using this program at some point.

Thanks, Alex