I have a archive handling library in TinyRange that supports automatically chaining handlers that cache intermediate steps. That is to say after a .tar.gz is downloaded it will write a seperate .tar file and a cache with a index of all the entries.
When your loading large archives (like the Linux source code) this substantially reduces loading time and it also means only the metadata needs to be kept in memory.
Vbitz
commented
3 months ago
I have a archive handling library in TinyRange that supports automatically chaining handlers that cache intermediate steps. That is to say after a
.tar.gzis downloaded it will write a seperate.tarfile and a cache with a index of all the entries.When your loading large archives (like the Linux source code) this substantially reduces loading time and it also means only the metadata needs to be kept in memory.