This is almost a requirement to enable pragzip for index creation in ratarmount instead of just index-reading. indexed_gzip already does something like this I think.
The advantage is that seeking latency inside a mounted archive would not vary the compression factor.
Furthermore, it would alleviate #2 because at least when we have a suitable index, the memory usage would also be independent of the compression ratio.
I think this would be easier to implement than #2 and therefore would be a good start. For starters, we could simply add all block boundaries to the Result type of the decoding routines. This would be a simple list of compressed/uncompressed offsets. The windows could be extracted from decompressed data, which is returned anyway. Then, the orchestrator thread can use all those seek points to find ideal ones and store them in the index database.
mxmlnkn
commented
1 year ago
This is almost a requirement to enable pragzip for index creation in ratarmount instead of just index-reading. indexed_gzip already does something like this I think.
I think this would be easier to implement than #2 and therefore would be a good start. For starters, we could simply add all block boundaries to the Result type of the decoding routines. This would be a simple list of compressed/uncompressed offsets. The windows could be extracted from decompressed data, which is returned anyway. Then, the orchestrator thread can use all those seek points to find ideal ones and store them in the index database.