search

airbreather / StepperUpper

Some tools for making STEP happen.
MIT License
9 stars 2 forks source link

Optimize 7-zip Archive Extraction #21

Open airbreather opened 7 years ago

airbreather commented 7 years ago

Currently, no matter how much of an archive file that the pack needs (or how little), StepperUpper extracts the entire 7-zip archive to disk. No matter how deeply the needed content is nested inside the archive, we also extract with the full paths, which can get extra lengthy and push the paths over the narrow Windows path length limit even if the finished product would fit inside that limit comfortably.

Ideally, we should be able to extract exactly the needed content to exactly its final locations.

One thing potentially making this difficult is that I don't know if there's a way to do this in a single invocation of 7-zip, which means that highly optimized archive files (like SMIM, Book of Silence, or the STEP Compilation) will have to re-inflate the same data many many times when installing something made up of many separate parts (like SMIM, Book of Silence, or the STEP Compilation).

airbreather commented 7 years ago

From Igor when someone else asked:

Command line 7-Zip doesn't support it now. Maybe it will be implemented in future.

So unless we use the API directly, there's no escaping the intermediate step to optimize path lengths.

We can, of course, have it limit which files get extracted, though.

airbreather commented 6 years ago

I've been looking into using SevenZipSharp for this, since it wraps the COM API and seems to offer something nice.

I'm not convinced it's going to fit in here, though. The asynchronous code is... weird at best. It needs to be really good if I'm going to start running all the archive extractions in-process. Alternatively, if I'm going to use it, but continue to farm the extraction tasks out to other processes, it needs to be really simple to use in order for me to justify that work.