search

CSB5 / OPERA-MS

OPERA-MS - Hybrid Metagenomic Assembler
Other
92 stars 17 forks source link

Error in during reference based clustering (line 35: 38236 Floating point exception(core dumped)) #89

Open ValeriiaLadyhina opened 9 months ago

ValeriiaLadyhina commented 9 months ago

Description of the problem

1) Question regarding memory: I ran

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat

on few different settings 1) 20 CPU 20G RAM, 2) 15 CPU 30G RAM 3) 10 CPU 50G RAM (have max 500 RAM) and I always get this Floating point exception(core dumped) . The error appears within max 40 seconds after I start the job.

Sizes of files that I am working with 11.31 Gbpls for paired Illumina and 2.113282Gbp nanopore reads (environment samples).

2)

Can you also try to run mash help command to see if this is not a compilation/system issue?

There is no problem with mash help.

jsgounot commented 9 months ago

Thank you Valeriia. This one will be difficult to resolve remotely. How big are the two mash files?

/proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh
/proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh

Would you be able to share the file when you have time? You can use wetransfer if the files are big. Those files are mash sketches (kmers subsampling of your genomes), with limited information (one could only guess what species are in your reads set).

If you can't share the file for any reason, there is not much I can offer you. As it's purely a mash issue, you could try to install a newer version of mash (with conda for example) and rerun the command line with a different executable. My intuition remains that there is not enough memory, but this is just an intuition. Mash pairwise comparison can be heavy memory-wise.

ValeriiaLadyhina commented 9 months ago

Good day!

1) genomes.msh is 112G 2) partial_sketch0.msh is 3.3M

I tried to install the newer version of mash, it didn't help.

If it is indeed a problem of memory, do you have a feeling how much of memory might be enough for my size of data?