mhx
commented
2 years ago I guess this is one of those cases where the feature sounds easy to implement, but it actually turns out to be quite hard.
There is, as usual, a trade-off here. If you're using something like -B30, you're entering an agreement at compression time that every small segment that makes up your file could be taken from any of the previous 30 file system blocks, each of which could be several MBs in size.
There are now a lot of possible ways to extract data from a DwarFS image. The way it's currently done is that you specify a range that you want to read from a file, e.g. you want to read 10,000 bytes from offset 25,0000. Then the DwarFS library determines which blocks it needs to decompress, decompresses them, and either passes you a list of chunks that you can iterate yourself or copies the chunks to a piece of memory. In the first case (list of chunks), this requires all filesystem blocks to be held in memory. In the second case, the blocks can be dismissed as soon as a chunk has been copied. The filesystem cache is used to keep the most recently accessed filesystem blocks around so they don't have to be decompressed over and over again. Now, dwarfsextract uses a pretty simple strategy: iterate over all files and for each file, read the full file and pass it to libarchive. This means that if you have huge files that potentially back-reference quite far into the list of blocks, dwarfsextract will consume a lot of memory.
A very memory efficient strategy that I can think of could be to decompress each filesystem block and slowly piece together the files created on-disk from their individual segments. There are a few drawbacks, though: first, this would be quite a bit of work to implement; second, in order to be efficient, this would require a reverse-lookup table that would potentially consume quite a bit of memory (though that table could be re-built per block); last but not least, this would likely not play well (or at all) with libarchive. So while it'd definitely be an interesting exercise, it'd be a lot of work and would likely make the implementation a lot more complex than it currently is.
One thing that should be quite trivial to implement and might get you pretty close to what you want would be to read individual files in blocks of a configurable size rather than reading them fully. This would definitely help if your use case is not only using -B30, but you're also storing huge files in the DwarFS filesystem.
Sorry for all the rambling, I've pretty much just wrote down my thoughts while pondering the problem. Can you confirm that you're running into this in cases where you're storing large files in the DwarFS filesystem?
adminx01
daci12345
This would be very useful. I dont want to give up on settings that are great at compressing such as -B30 but at extraction it can use a ton of RAM depending on the type of file.