Closed sfiligoi closed 1 year ago
On EMP-style BIOM, the new without_replacement algorithm is about 2x faster (n=1000, on a EPYC 7302 CPU): 33s vs 58s
There is also a small reduction is memory consumption, but it is barely noticeable compared to the rest of the memory use when using the Table object.
For the record, the biom used for testing was mp.90.min25.deblur.withtax.onlytree_ACTUAL_overlap.biom
For BIOM tables with very large per-column sums, it is an enabler; the old code would outright fail if the columns sum was > 2^31. (e.g. rna_integrity_metaG_woltka_wolr2_with_plasmids_per_gene_clean_gene_counts_per_g_stool.biom)
(But just fixing that would not help... a test showed it would have needed over 15.1 TiB of RAM)
The new code has a 2^63 limit, and the memory use is proportional to n, only.
The running time is quite fast for small n, but gets slower with high n (on same CPU as above): n=2000 50s n=6G 20min
Thanks!!
Drastically reduces the memory needs when sums are large. Also allows sums to be >2^31.