state of kff-tools on human-scale data

[rchikhi]

Test setup: human reads from https://www.ncbi.nlm.nih.gov/sra/?term=SRR034956 158G all.fastq.gz ran KMC to produce a KFF file:

\time kmc -fq -okff -k31 -ci2  all.fastq.gz kmc_k31 .
   Total no. of k-mers                :  94768162610
   Total no. of reads                 :   1415483596
   Total no. of super-k-mers          :   9005121051

33G kmc_k31.kff

status	tool	time	memory
☑️	kmc	41m	11 GB
☑️	kff-tools outstr	2h25	< 1 MB
☑️	kff-tools instr	40m	< 1 MB
☑️	kff-tools translate	5m	< 1 MB
☑️	kff-tools data-rm	5m	< 1 MB
☑️	kff-tools validate	3m	< 1 MB
☑️	kff-tools disjoin	9m	< 1 MB
☑️	kff-tools split	48m	< 1 MB
☑️	kff-tools merge	1h	< 1 MB
☑️	kff-tools sort	28m	1.4 GB
☑️	kff-tools shuffle	47m	1.4 GB
❌	kff-tools bucket

Remarks:

• outstr slower than it needs to be
• bucket crashes with signal 11 (kff-tools bucket --infile kmc_k31.kff -o test_bucket -m 10)

[yoann-dufresne]

Bucket tool currently under optimization (bucket branch)

[yoann-dufresne]

bucket is now working on Human genome Compact is still too slow