Closed marcelm closed 10 months ago
I think its a great idea to unconditionally set it to use 64 bit bucket start indices for now and make a release so that https://github.com/ksahlin/strobealign/issues/277 and https://github.com/ksahlin/strobealign/issues/285 gets resolved. Let's do that!
As you say, going from 1gb to 2gb in human or 2gb to 4gb in rye) is negligible compared to flat vector memory.
When we met yesterday, we decided to close this issue for now as we think it is quite some work to dynamically switch between 32 and 64 bit indices. Always using 64 bits is good enough. Also, we are considering some further memory reduction, which are no longer possible if the index vector only uses 32 bits.
Here is my to-do list for making strobealign work with references with more than $2^{32}$ strobemers. This will resolve #277 and #285.
unsigned
touint64_t
. This can be done unconditionally as it’s not a memory or performance issue when these are just always 64 bit.int
andunsigned
and check whether they need to be replace withbucket_index_t
,size_t
,uint64_t
bucket_index_t
typedef touint64_t
StrobemerIndex
into a separate class (let’s call itHashtable
here)bucket_index_t
a template parameter of that classHashtable<uint32_t>
orHashtable<uint64_t>
(use a virtual function or a function pointer).Much of the above is done already and the question for me is whether we perhaps want to release a strobealign version now that unconditionally uses 64 bit bucket start indices because I see that it is a bit of work to switch to dynamically deciding which size of indices to use.
For CHM13, the index vector currently needs 1 GiB ($2^{28}$ entries times 4 bytes per entry). That size would double with 64-bit indices, so 2 GiB. Overall memory usage would increase from 13 GiB to 14 GiB. But then, memory usage was 22 GiB before merging #278, so the savings are still huge.