search

faircloth-lab / phyluce

software for UCE (and general) phylogenomics
http://phyluce.readthedocs.org/
Other
80 stars 49 forks source link

issue spades: Expected assembly files were not found in output. #240

Closed haxacloruro closed 3 years ago

haxacloruro commented 3 years ago

Hi, I used spades to assembly my data and I successfully did it in all my samples, but not in five. I'm having a warning that says "Expected assembly files were not found in output". At the end I have empty contig files from those samples. I do not understand why in this group of samples Spades is working for some samples but not for all. I wonder if you could have an idea what is happening. Best!

2021-07-20 11:30:40,319 - phyluce_assembly_assemblo_spades_tmp - INFO - ========= Starting phyluce_assembly_assemblo_spades_tmp ========= 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Version: 1.7.1 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Commit: None 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --config: /ddnB/work/habromys/U1/spades/spades115.conf 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --cores: 16 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --dir: None 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --log_path: /work/habromys/U1/spades/spades115_spa 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --memory: 32 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --no_clean: False 2021-07-20 11:30:40,320 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --output: /ddnB/work/habromys/U1/spades/spades115_spa 2021-07-20 11:30:40,321 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --subfolder: 2021-07-20 11:30:40,321 - phyluce_assembly_assemblo_spades_tmp - INFO - Argument --verbosity: INFO 2021-07-20 11:30:40,321 - phyluce_assembly_assemblo_spades_tmp - INFO - Getting input filenames and creating output directories 2021-07-20 11:30:40,347 - phyluce_assembly_assemblo_spades_tmp - INFO - ------------- Processing MZFC13091_Peromyscus_beatae ------------ 2021-07-20 11:30:40,348 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 11:30:40,353 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 11:30:40,354 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 11:36:35,418 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 11:36:35,420 - phyluce_assembly_assemblo_spades_tmp - CRITICAL - Expected assembly files were not found in output. 2021-07-20 11:36:35,421 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 11:36:35,421 - phyluce_assembly_assemblo_spades_tmp - INFO - ---------- Processing MZFC9736_Peromyscus_carolpattonae --------- 2021-07-20 11:36:35,422 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 11:36:35,426 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 11:36:35,426 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 11:44:08,398 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 11:44:08,401 - phyluce_assembly_assemblo_spades_tmp - CRITICAL - Expected assembly files were not found in output. 2021-07-20 11:44:08,401 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 11:44:08,402 - phyluce_assembly_assemblo_spades_tmp - INFO - -------- Processing MZFC10521_Peromyscus_gratus_gentilis -------- 2021-07-20 11:44:08,403 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 11:44:08,407 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 11:44:08,407 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 11:50:52,040 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 11:50:52,043 - phyluce_assembly_assemblo_spades_tmp - CRITICAL - Expected assembly files were not found in output. 2021-07-20 11:50:52,043 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 11:50:52,044 - phyluce_assembly_assemblo_spades_tmp - INFO - ----------- Processing MZFC15046_Peromyscus_spicilegus ---------- 2021-07-20 11:50:52,044 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 11:50:52,049 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 11:50:52,049 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 12:30:30,717 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 12:30:30,901 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 12:30:30,902 - phyluce_assembly_assemblo_spades_tmp - INFO - ------- Processing MZFC12350_Reithrodontomys_fulvescens_2 ------- 2021-07-20 12:30:30,902 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 12:30:30,906 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 12:30:30,906 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 13:22:28,227 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 13:22:28,351 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 13:22:28,352 - phyluce_assembly_assemblo_spades_tmp - INFO - ------------ Processing MZFC7841_Scotinomys_teguina_3 ----------- 2021-07-20 13:22:28,352 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 13:22:28,356 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 13:22:28,356 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 15:11:23,499 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 15:11:23,595 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 15:11:23,596 - phyluce_assembly_assemblo_spades_tmp - INFO - ------------ Processing MZFC11015_Habromys_schmidlyi ------------ 2021-07-20 15:11:23,596 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 15:11:23,600 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 15:11:23,600 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 15:18:34,027 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 15:18:34,029 - phyluce_assembly_assemblo_spades_tmp - CRITICAL - Expected assembly files were not found in output. 2021-07-20 15:18:34,029 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 15:18:34,030 - phyluce_assembly_assemblo_spades_tmp - INFO - ----------- Processing MZFC11166_Peromyscus_mexicanus ----------- 2021-07-20 15:18:34,031 - phyluce_assembly_assemblo_spades_tmp - INFO - Finding fastq/fasta files 2021-07-20 15:18:34,035 - phyluce_assembly_assemblo_spades_tmp - INFO - File type is fastq 2021-07-20 15:18:34,035 - phyluce_assembly_assemblo_spades_tmp - INFO - Running SPAdes for PE data 2021-07-20 15:27:35,672 - phyluce_assembly_assemblo_spades_tmp - INFO - Removing extraneous assembly files 2021-07-20 15:27:35,674 - phyluce_assembly_assemblo_spades_tmp - CRITICAL - Expected assembly files were not found in output. 2021-07-20 15:27:35,674 - phyluce_assembly_assemblo_spades_tmp - INFO - Symlinking assembled contigs into /ddnB/work/habromys/U1/spades/spades115_spa/contigs 2021-07-20 15:27:35,675 - phyluce_assembly_assemblo_spades_tmp - INFO - ========= Completed phyluce_assembly_assemblo_spades_tmp ========

GabiCamacho commented 3 years ago

I am having the same issue. Have you figure this out already?

haxacloruro commented 3 years ago

no yet...

brantfaircloth commented 3 years ago

It is very likely that Spades is running out of RAM to assemble the reads you have for the samples that are not assembling. There are two potential ways to fix this issue:

  1. Find a machine with sufficient RAM and set the --memory parameter to ~ 4 GB below the amount of RAM on this machine, e.g. --memory 60. Or even better, --memory 250. The value you use is the number, in GB, of RAM to allocate to Spades.

  2. Alternatively, you can downsample the reads you have for the individuals that are failing to assemble, using an approach like this. I usually downsample to something between 3 million reads (in total; so 1.5 million in R1 and 1.5 million in R2) and 6 million reads.

haxacloruro commented 3 years ago

I had not not clarify the allowed memory paramter. It worked after I did. Thank you.