Errors from reference-guided mode, using example data

[ctseto]

From the example data: -- I have a set of metagenomic reads, and want to perform reference-guided assembly.

python3 go_metacompass.py -P [read1.fq,read2.fq] -l [max read length]-o [output_folder] -m [min coverage] -t [ncpu]

python3 go_metacompass.py -P tutorial/thao2000.1.fq,tutorial/thao2000.2.fq -l 150 -o example1_output_referenceguidedassembly -m 1 -t 4

Error: ... ---Build index . Full Traceback (most recent call last): File "/home/cseto/Programs/anaconda3/envs/MetaCompass/lib/python3.5/site-packages/snakemake/executors.py", line 784, in run_wrapper version) File "/home/cseto/Programs/MetaCompass-1.12/snakemake/metacompass.iter0.paired.py", line 337, in rule_bowtie2_map File "/home/cseto/Programs/anaconda3/envs/MetaCompass/lib/python3.5/site-packages/snakemake/shell.py", line 74, in new__ raise sp.CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'bowtie2-build -o 3 --threads 4 -q example1_output_referenceguidedassembly/thao2000.0.assembly.out/mc.refseq.filt.fna example1_output_referenceguidedassembly/thao2000.0.assembly.out/thao2000.index 1>> example1_output_referenceguidedassembly/thao2000.0.assembly.out/thao2000.index 2>&1;bowtie2 -a --end-to-end --sensitive --no-unal -p 4 -x example1_output_referenceguidedassembly/thao2000.0.assembly.out/thao2000.index -q -U example1_output_referenceguidedassembly/thao2000.merged.fq -S example1_output_referenceguidedassembly/thao2000 .0.assembly.out/thao2000.sam.all > example1_output_referenceguidedassembly/thao2000.0.bowtie2map.log 2>&1; /home/cseto/Programs/MetaCompass-1.12/bin/best_strata.py example1_output_referencegu idedassembly/thao2000.0.assembly.out/thao2000.sam.all example1_output_referenceguidedassembly/thao2000.0.assembly.out/thao2000.sam; rm example1_output_referenceguidedassembly/thao2000.0.assembly.out/thao2000.sam.all' returned non-zero exit status 1

Presumably this begins at bowtie2-build -o 3 --threads 4 -q example1_output_referenceguidedassembly/thao2000.0.assembly.out/mc.refseq.filt.fna example1_output_referenceguidedass embly/thao2000.0.assembly.out/thao2000.index Warning: Empty fasta file: 'example1_output_referenceguidedassembly/thao2000.0.assembly.out/mc.refseq.filt.fna' Warning: All fasta inputs were empty Error: Encountered internal Bowtie 2 exception (#1)

However, assembly with reference (-r) seems to work fine.

[vcepeda]

Please use the last release 1.2 for now. I'm currently working on the new 1.3 version that will be released soon.

[bharat1912]

1. Ran the SRS data files and it errored out (metacompass) bharat@bharat-T3500:~/opt/anaconda3/envs/metacompass/MetaCompass$ python3 go_metacompass.py -P tutorial/SRS044742/SRS044742.denovo_duplicates_marked.trimmed.1.fastq,tutorial/SRS044742/SRS044742.denovo_duplicates_marked.trimmed.2.fastq -U tutorial/SRS044742/SRS044742.denovo_duplicates_marked.trimmed.singleton.fastq -l 100 -o tutorial/example2_output

checking for dependencies (Bowtie2, Blast, kmermask, Snakemake, etc) /home/bharat/opt/anaconda3/envs/metacompass/bin/snakemake Snakemake--->[OK] Bowtie2--->[OK] /home/bharat/opt/anaconda3/envs/metacompass/bin/blastn Blast+--->[OK] /home/bharat/opt/anaconda3/envs/metacompass/bin/kmer-mask kmer-mask--->[OK] /usr/bin/mash mash--->[OK] Provided cores: 1 Rules claiming more threads will be scaled down. Job counts: count jobs 1 all 1 assemble_unmapped 1 bam_sort 1 bowtie2_map 1 build_contigs 1 create_tsv 1 fastq2fasta 1 join_contigs 1 kmer_mask 1 merge_reads 1 pilon_contigs 1 pilon_map 1 reference_recruitment 1 sam_to_bam 14 Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): merge_reads Selected jobs (1): merge_reads Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 12: ---merge fastq reads Reason: Missing output files: tutorial/example2_output/SRS044742.merged.fq

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 12. 1 of 14 steps (7%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): kmer_mask Selected jobs (1): kmer_mask Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 13: ---kmer-mask fastq Reason: Missing output files: tutorial/example2_output/SRS044742.marker.match.1.fastq; Input files updated by another job: tutorial/example2_output/SRS044742.merged.fq

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 13. 2 of 14 steps (14%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): fastq2fasta Selected jobs (1): fastq2fasta Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 11: ---Converting fastq to fasta. Reason: Missing output files: tutorial/example2_output/SRS044742.fasta; Input files updated by another job: tutorial/example2_output/SRS044742.marker.match.1.fastq

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 11. 3 of 14 steps (21%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): reference_recruitment Selected jobs (1): reference_recruitment Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 8: ---reference recruitment. Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/mc.refseq.fna; Input files updated by another job: tutorial/example2_output/SRS044742.fasta, tutorial/example2_output/SRS044742.merged.fq

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 8. 4 of 14 steps (29%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): bowtie2_map Selected jobs (1): bowtie2_map Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 10: ---Build index . Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.sam; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/mc.refseq.fna, tutorial/example2_output/SRS044742.merged.fq

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 10. 5 of 14 steps (36%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): build_contigs Selected jobs (1): build_contigs Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 6: ---Build contigs . Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/mc.refseq.fna, tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.sam

Skipped removing non-empty directory tutorial/example2_output/SRS044742.0.assembly.out Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 6. 6 of 14 steps (43%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): pilon_map Selected jobs (1): pilon_map Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 7: ---Map reads for pilon polishing. Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam.unmapped.2.fq, tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam.unmapped.1.fq, tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc_unpaired.sam, tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 7. 7 of 14 steps (50%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (2): assemble_unmapped sam_to_bam Selected jobs (1): sam_to_bam Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 9: ---Convert sam to bam . Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam.bam; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam, tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc_unpaired.sam

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 9. 8 of 14 steps (57%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (2): assemble_unmapped bam_sort Selected jobs (1): bam_sort Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 5: ---Sort bam . Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/sorted.bam, tutorial/example2_output/SRS044742.0.assembly.out/sorted2.bam; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam.bam

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 5. 9 of 14 steps (64%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (2): assemble_unmapped pilon_contigs Selected jobs (1): assemble_unmapped Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 3: ---Assemble unmapped reads . Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.megahit/final.contigs.fa; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam.unmapped.2.fq, tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.mc.sam.unmapped.1.fq

Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Finished job 3. 10 of 14 steps (71%) done Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} Ready jobs (1): pilon_contigs Selected jobs (1): pilon_contigs Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 2: ---Pilon polish contigs . Reason: Missing output files: tutorial/example2_output/SRS044742.0.assembly.out/contigs.pilon.fasta; Input files updated by another job: tutorial/example2_output/SRS044742.0.assembly.out/sorted.bam, tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta, tutorial/example2_output/SRS044742.0.assembly.out/sorted2.bam

Full Traceback (most recent call last): File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 923, in run_wrapper log, version, rule, conda_env, None) File "/home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/snakemake/metacompass.iter0.py", line 564, in rule_pilon_contigs File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/site-packages/snakemake/shell.py", line 88, in new__ raise sp.CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'java -Xmx19G -jar /home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/bin/pilon-1.23.jar --flank 5 --threads 1 --mindepth 3 --genome tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta --frags tutorial/example2_output/SRS044742.0.assembly.out/sorted.bam --unpaired tutorial/example2_output/SRS044742.0.assembly.out/sorted2.bam --output tutorial/example2_output/SRS044742.0.assembly.out/contigs.pilon --fix bases,amb --tracks --changes --vcf 1>> tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.pilon.log 2>&1' returned non-zero exit status 1.

Error in job pilon_contigs while creating output file tutorial/example2_output/SRS044742.0.assembly.out/contigs.pilon.fasta. Full Traceback (most recent call last): File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 923, in run_wrapper log, version, rule, conda_env, None) File "/home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/snakemake/metacompass.iter0.py", line 564, in rule_pilon_contigs File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/site-packages/snakemake/shell.py", line 88, in new__ raise sp.CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'java -Xmx19G -jar /home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/bin/pilon-1.23.jar --flank 5 --threads 1 --mindepth 3 --genome tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta --frags tutorial/example2_output/SRS044742.0.assembly.out/sorted.bam --unpaired tutorial/example2_output/SRS044742.0.assembly.out/sorted2.bam --output tutorial/example2_output/SRS044742.0.assembly.out/contigs.pilon --fix bases,amb --tracks --changes --vcf 1>> tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.pilon.log 2>&1' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 326, in _callback raise ex File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 935, in run_wrapper show_traceback=True)) snakemake.exceptions.RuleException: CalledProcessError in line 244 of /home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/snakemake/metacompass.iter0.py: Command 'java -Xmx19G -jar /home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/bin/pilon-1.23.jar --flank 5 --threads 1 --mindepth 3 --genome tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta --frags tutorial/example2_output/SRS044742.0.assembly.out/sorted.bam --unpaired tutorial/example2_output/SRS044742.0.assembly.out/sorted2.bam --output tutorial/example2_output/SRS044742.0.assembly.out/contigs.pilon --fix bases,amb --tracks --changes --vcf 1>> tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.pilon.log 2>&1' returned non-zero exit status 1. File "/home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/snakemake/metacompass.iter0.py", line 244, in __rule_pilon_contigs

RuleException: CalledProcessError in line 244 of /home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/snakemake/metacompass.iter0.py: Command 'java -Xmx19G -jar /home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/bin/pilon-1.23.jar --flank 5 --threads 1 --mindepth 3 --genome tutorial/example2_output/SRS044742.0.assembly.out/contigs.fasta --frags tutorial/example2_output/SRS044742.0.assembly.out/sorted.bam --unpaired tutorial/example2_output/SRS044742.0.assembly.out/sorted2.bam --output tutorial/example2_output/SRS044742.0.assembly.out/contigs.pilon --fix bases,amb --tracks --changes --vcf 1>> tutorial/example2_output/SRS044742.0.assembly.out/SRS044742.pilon.log 2>&1' returned non-zero exit status 1. File "/home/bharat/opt/anaconda3/envs/metacompass/MetaCompass/snakemake/metacompass.iter0.py", line 244, in __rule_pilon_contigs File "/home/bharat/opt/anaconda3/envs/metacompass/lib/python3.6/concurrent/futures/thread.py", line 56, in run Releasing 1 _cores (now 1). Releasing 1 _nodes (now 9223372036854775807). Will exit after finishing currently running jobs. Exiting because a job execution failed. Look above for error message unlocking removing lock removing lock removed all locks ERROR: snakemake command failed; exiting..

2. The outputs from the run is give below: (metacompass) bharat@bharat-T3500:~/opt/anaconda3/envs/metacompass/MetaCompass/tutorial/example2_output$ ls -l -a total 218604 drwxr-xr-x 3 bharat bharat 4096 Jul 28 18:41 . drwxr-xr-x 4 bharat bharat 4096 Jul 28 18:38 .. drwxr-xr-x 3 bharat bharat 4096 Jul 28 18:46 SRS044742.0.assembly.out -rw-r--r-- 1 bharat bharat 213 Jul 28 18:40 SRS044742.0.bowtie2map.log -rw-r--r-- 1 bharat bharat 64 Jul 28 18:39 SRS044742.0.kmermask.log -rw-r--r-- 1 bharat bharat 3500 Jul 28 18:45 SRS044742.0.megahit.log -rw-r--r-- 1 bharat bharat 562 Jul 28 18:41 SRS044742.0.pilon.map.log -rw-r--r-- 1 bharat bharat 282103 Jul 28 18:39 SRS044742.0.reference_recruitement.log -rw-r--r-- 1 bharat bharat 605431 Jul 28 18:39 SRS044742.fasta -rw-r--r-- 1 bharat bharat 0 Jul 28 18:38 SRS044742.marker.clean.1.fastq -rw-r--r-- 1 bharat bharat 1133498 Jul 28 18:39 SRS044742.marker.match.1.fastq -rw-r--r-- 1 bharat bharat 0 Jul 28 18:38 SRS044742.marker.mixed.1.fastq -rw-r--r-- 1 bharat bharat 114360182 Jul 28 18:39 SRS044742.marker.murky.1.fastq -rw-r--r-- 1 bharat bharat 107435686 Jul 28 18:38 SRS044742.merged.fq

3. python3 go_metacompass.py -r tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta -P tutorial/thao2000.1.fq,tutorial/thao2000.2.fq -l 150 -o example1_output -m 1 -t 4

This completed without any problems.

[vcepeda]

How much memory do you have? Pilon needs 19G to run.

[bharat1912]

How much memory do you have? Pilon needs 19G to run.

Thanks. It was a 12G computer. I have since installed and moved to a newer machine with 64G; both the examples have run and exited with no errors.

Many thanks for your prompt assistance. Much appreciated.