search

mstange / symsrv

Download pdbs from symbol servers and cache locally, parse symbol paths from env vars
Apache License 2.0
21 stars 2 forks source link

CAB decompression should happen in a streaming fashion #6

Closed mstange closed 6 months ago

mstange commented 1 year ago

At the moment, for CAB-compressed files, we load the entire file into memory, then uncompress the entire thing, and then write it to disk.

We should do the decompression per chunk.

mstange commented 1 year ago

This is actually harder than I initially realized. The cab crate currently requires the input to implement Read + Seek. I think we might need to fork it to use AsyncRead and limit its use to cab files which never need to seek backwards.

For each item in a cab archive, the file contains a table which maps between compressed input file offsets and uncompressed output file offset ranges. These offsets could, in general, jump back and forth in the input file. But my expectation is that for most files they'll only move forwards.

mstange commented 6 months ago

This is implemented.