search

morispi / CONSENT

Scalable long read self-correction and assembly polishing with multiple sequence alignment
https://doi.org/10.1038/s41598-020-80757-5
GNU Affero General Public License v3.0
55 stars 5 forks source link

CONSENT-polish fails at sort step #15

Open nadegeguiglielmoni opened 4 years ago

nadegeguiglielmoni commented 4 years ago

CONSENT v1.2.2

Hi,

Thank you for this tool.

I'm struggling to use CONSENT on a cluster, and the latest error I get is at the sort step :

Error log

[M::main] Version: 2.17-r941 [M::main] CMD: /home/nguiglie/Tools/minimap2-2.17_x64-linux/minimap2 --dual=yes -PD --no-long-join -w5 -g1000 -m30 -n1 -t30 -I1G raven_default.min100kb.assembly_1.fasta reads.pacbio.01.fasta [M::main] Real time: 7419.892 sec; CPU: 179145.556 sec; Peak RSS: 18.149 GB sort: erreur d'écriture: Erreur d'entrée/sortie

Output log

[sam. nov. 23 23:20:46 CET 2019] Aligning the long reads to the contigs (minimap2) [dim. nov. 24 01:24:34 CET 2019] Sorting the overlaps

Sort version: sort (GNU coreutils) 8.22

Infos about output files:

124G 24 nov. 01:47 exploded_66171_2 121G 24 nov. 02:08 exploded_66171_3 115G 24 nov. 04:00 sorted_exploded_66171_1 19G 24 nov. 06:01 sorted_exploded_66171_2 359G 24 nov. 01:24 tmp_Alignments_66171.paf

Have you got any idea what could be the problem? Also, it would be convenient to be able to restart the program.

morispi commented 4 years ago

Hi Nadège,

Sorry for answering so late, I'm quite busy with teaching lately.

I believe the problem might be that you used more memory than your cluster allows you to? I mean, I'm not sure how your cluster is set up, but judging from the error (sort not completing its job properly) I don't really know what else it could be. Googling the issue with sort didn't seem to help much...

Could you maybe try to run CONSENT-polish on a smaller dataset? That could help narrowing down the problem, and see if it's actually caused by an excessive usage of disk space.

Also, great suggestion, restarting the program from the last step would be super convenient, I'll be sure to include it in a further release.

Cheers, Pierre

sadikmu commented 4 years ago

Hi,

CONSENT-polish failed at sort step. Any suggestions please?

CONSENT-polish --contigs contigs.fasta --reads epo.pacbio.fasta.gz --out polished.fasta

Last lines of err.out 

[M::main] Version: 2.17-r941
[M::main] CMD: minimap2 --dual=yes -PD --no-long-join -w5 -g1000 -m30 -n1 -t88 -I1G contigs.fasta pacbio.fasta.gz
[M::main] Real time: 25939.646 sec; CPU: 404591.600 sec; Peak RSS: 37.921 GB
sort: close failed: ./sort2wwjKa: Input/output error

Thanks