search

jkbonfield / fqzcomp

Fastq compression tool
14 stars 0 forks source link

Crash with -s9 #2

Open KirillKryukov opened 4 years ago

KirillKryukov commented 4 years ago

fqzcomp -s9 crashes during compression.

bolt% fqzcomp -s1 -e -q1 -n2 <a.fastq >a.fastq.fqzcomp-s1
Names      61229 ->         65 (0.001)
Bases    1446700 ->     361472 (0.250)
Quals    1446700 ->        432 (0.000)
bolt% fqzcomp -s8 -e -q1 -n2 <a.fastq >a.fastq.fqzcomp-s8
Names      61229 ->         65 (0.001)
Bases    1446700 ->     360638 (0.249)
Quals    1446700 ->        432 (0.000)
bolt% fqzcomp -s9 -e -q1 -n2 <a.fastq >a.fastq.fqzcomp-s9
zsh: segmentation fault (core dumped)  fqzcomp -s9 -e -q1 -n2 < a.fastq > a.fastq.fqzcomp-s9

Tried with various data, -s9 always crashes, even with tiny inputs.

Let me know if you need test data or a complete repro srcipt.

keiranmraine commented 2 years ago

I can confirm this behaviour, once memory was pushed high enough, I began to see the crash rather than oom.

jkbonfield commented 2 years ago

Hmm, I can reproduce this too. I'm away for a bit, but will try and diagnose the problem later.

Realistically though it's all diminishing returns and maybe -s8 is a better maximum anyway.

keiranmraine commented 2 years ago

I agree, especially when memory is taken into account for 2 CPUs, -s8 is ~8GB, it took >30GB to get -s9 to run to the failure point.