search

ablab / spades

SPAdes Genome Assembler
http://ablab.github.io/spades/
Other
737 stars 134 forks source link

Error 20 #380

Open szimmerman92 opened 4 years ago

szimmerman92 commented 4 years ago

Hi,

When I run spades for a group of samples I keep on getting the following error mesages

======= SPAdes pipeline started. Log can be found here: /n/scratchlfs/buenrostro_lab/sam_rotation/assemblies/SRR5983368_ms_out/spades.log

===== Read error correction started.

error listing files in '/n/scratchlfs/buenrostro_lab/sam_rotation/assemblies/SRR5983368_ms_out/configs/hammer': Not a directory Traceback (most recent call last): File "/n/home13/szimmerman/miniconda3/envs/meta_assemblers/lib/python3.7/distutils/dir_util.py", line 126, in copy_tree names = os.listdir(src) NotADirectoryError: [Errno 20] Not a directory: '/n/scratchlfs/buenrostro_lab/sam_rotation/assemblies/SRR5983368_ms_out/configs/hammer'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/n/home13/szimmerman/SPAdes-3.13.1-Linux/bin/metaspades.py", line 797, in main ext_python_modules_home, only_compressing_is_needed, log) File "/n/home13/szimmerman/SPAdes-3.13.1-Linux/share/spades/spades_pipeline/hammer_logic.py", line 131, in run_hammer dir_util.copy_tree(os.path.join(configs_dir, "hammer"), dst_configs, preserve_times=False) File "/n/home13/szimmerman/miniconda3/envs/meta_assemblers/lib/python3.7/distutils/dir_util.py", line 132, in copy_tree "error listing files in '%s': %s" % (src, e.strerror)) distutils.errors.DistutilsFileError: error listing files in '/n/scratchlfs/buenrostro_lab/sam_rotation/assemblies/SRR5983368_ms_out/configs/hammer': N ot a directory

== Error == exception caught: <class 'distutils.errors.DistutilsFileError'>

In case you have troubles running SPAdes, you can write to spades.support@cab.spbu.ru or report an issue on our GitHub repository github.com/ablab/spades Please provide us with params.txt and spades.log files from the output directory.

However when I go to the directory, /n/scratchlfs/buenrostro_lab/sam_rotation/assemblies/SRR5983368_ms_out/configs/hammer

it definitely exists.

Attached you can find the params and log files.

Thank you very much!

Best, Sam params.txt spades.log

asl commented 4 years ago

Hello

This really looks like an I/O problem on your end. Can you try to move SPAdes output dir to some other volume / disk?

szimmerman92 commented 4 years ago

Definitely. I will try and let you know how it goes. Thank you very much!

szimmerman92 commented 4 years ago

Spades is no longer crashing anymore, but spades seems to be either stuck, or is taking an extremely long time. So far it has been running for about 13 hours. I have attached the log and parameter files. Below is the output. Could there be a problem with the location I am storing the fastq files?

params.txt spades.log

Thank you!

== Running read error correction tool: /n/home13/szimmerman/SPAdes-3.13.1-Linux/bin/spades-hammer /n/home13/szimmerman/remaining_ebi_assemblies/SRR2155377_ms_out/corrected/configs/config.info

0:00:00.000 4M / 4M INFO General (main.cpp : 75) Starting BayesHammer, built from refs/heads/spades_3.13.1, git revision 9a9d54db2ff9abaac718155 bf74c12ec9464e8ca 0:00:00.001 4M / 4M INFO General (main.cpp : 76) Loading config from /n/home13/szimmerman/remaining_ebi_assemblies/SRR2155377_ms_out/corrected/c onfigs/config.info 0:00:00.006 4M / 4M INFO General (main.cpp : 78) Maximum # of threads to use (adjusted due to OMP capabilities): 7 0:00:00.006 4M / 4M INFO General (memory_limit.cpp : 49) Memory limit set to 250 Gb 0:00:00.006 4M / 4M INFO General (main.cpp : 86) Trying to determine PHRED offset 0:00:00.025 4M / 4M INFO General (main.cpp : 92) Determined value is 33 0:00:00.026 4M / 4M INFO General (hammer_tools.cpp : 36) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ] 0:00:00.026 4M / 4M INFO General (main.cpp : 113) Size of aux. kmer data 24 bytes === ITERATION 0 begins === 0:00:00.026 4M / 4M INFO K-mer Counting (kmer_data.cpp : 280) Estimating k-mer count 0:00:00.069 116M / 116M INFO K-mer Counting (kmer_data.cpp : 285) Processing /n/scratchlfs/buenrostro_lab/sam_rotation/no_human_ebi/SRR2155377_no_human_reads/SRR 2155377_human_free_1.fastq.gz 0:00:45.694 140M / 140M INFO K-mer Counting (kmer_data.cpp : 294) Processed 11988866 reads 0:00:45.695 140M / 140M INFO K-mer Counting (kmer_data.cpp : 285) Processing /n/scratchlfs/buenrostro_lab/sam_rotation/no_human_ebi/SRR2155377_no_human_reads/SRR 2155377_human_free_2.fastq.gz 0:01:28.678 140M / 140M INFO K-mer Counting (kmer_data.cpp : 294) Processed 23977732 reads 0:01:28.678 140M / 140M INFO K-mer Counting (kmer_data.cpp : 299) Total 23977732 reads processed 0:01:30.157 140M / 140M INFO K-mer Counting (kmer_data.cpp : 306) Estimated 529219866 distinct kmers 0:01:30.157 28M / 140M INFO K-mer Counting (kmer_data.cpp : 311) Filtering singleton k-mers 38 8 0 nslots: 1073741824 bits per slot: 8 range: 0000004000000000 0:01:30.157 1G / 1G INFO K-mer Counting (kmer_data.cpp : 317) Processing /n/scratchlfs/buenrostro_lab/sam_rotation/no_human_ebi/SRR2155377_no_human_reads/SRR 2155377_human_free_1.fastq.gz 0:10:26.343 1G / 1G INFO K-mer Counting (kmer_data.cpp : 326) Processed 11988866 reads 0:10:26.344 1G / 1G INFO K-mer Counting (kmer_data.cpp : 317) Processing /n/scratchlfs/buenrostro_lab/sam_rotation/no_human_ebi/SRR2155377_no_human_reads/SRR 2155377_human_free_2.fastq.gz

asl commented 4 years ago

Unfortunately, this is a known problem with CQF implementation we're using. There is no workaround currently. You may try to quality trim your read and run --only-assembler

szimmerman92 commented 4 years ago

Thank you!

alienzj commented 3 years ago

@asl Hi, what is CQF implementation ? Can I contribute it ?

asl commented 3 years ago

@alienzj Sure, patches are welcome!

alienzj commented 3 years ago

@asl Hi, what is CQF ? full name ?

asl commented 3 years ago

"counting quotient filter"

alienzj commented 3 years ago

Thanks, I'll dig into it.