mxmlnkn / rapidgzip

Gzip Decompression and Random Access for Modern Multi-Core Machines
Apache License 2.0
345 stars 7 forks source link

Access large indexes directly from the file system #10

Open mxmlnkn opened 1 year ago

mxmlnkn commented 1 year ago

gztool has something like this:

windows are not loaded in memory unless they're needed, so the application memory footprint is fairly low

I think it should be fairly easy to implement thanks to the WindowMap class abstraction access to windows. I might have to make it multi-threading safe though. On Linux systems, this abstraction could simply use mmap to implement it.

This also would improve startup time and interleave index loading with decompression.

I think it only makes sense for indexes that are larger than ~32 MiB. Because the memory usage for decompression should at least be in this order.