Decompress with ISA-L - Githubissues

ksahlin / strobealign

Aligns short reads using dynamic seed size with strobemers

MIT License

128 stars 16 forks source link

Open marcelm opened 2 months ago

marcelm commented 2 months ago

This is a continuation of #269 where I fixed the things I commented on.

I will need to re-measure memory usage. At some point, it was higher than previously, but that seems to be fixed now.

As intended, runtime improves slightly.

Marked as draft because two issues remain:

[ ] Memory usage increases significantly, probably because of mmap usage, which I don’t think makes so much sense.
[ ] Multi-block gzips (concatenated gzip files) are not supported and hang the input thread.

marcelm commented 2 months ago

I found two issues:

Memory usage increases significantly, probably because of mmap usage. I don’t think using mmap makes a lot of sense because the input should be streamed instead.
Multi-block gzips (concatenated gzip files) are not supported and hang the input thread.

I’ll leave this for now, there are bigger improvements to be made elsewhere.

teepean commented 2 months ago

Htslib uses libdeflate so maybe that would be a better option?

marcelm commented 2 months ago

The Python isal bindings support multiblock gzips, so it is doable with ISA-L (maybe only a newer version is needed).