Use SharpCompress for Unpack

[takano32]

closes #39, closes #38

SharpCompress is a compression library in pure C#.

Using this make Unpack available in Linux and macOS.

[takano32]

I am debugging a problem where the percentage values are somehow wrong, but I am not sure.

[takano32]

The progress indicator is odd, but it works very well.

After all, it is a very nice experience to be able to convert files that are still compressed on Linux and macOS.

So I really want to get rid of this strange bug, but it may be difficult to get rid of it on my own.

Would you have any idea or knowledge for archiver?

Around here: https://github.com/RupertAvery/PSXPackager/pull/40/files#diff-6cfd5512bb6cd309723f143eadffb66f0536d7773dc6080fa14939ba30eec779R343-R348

Now I decided to make this PR from Draft to Ready for Review, but I’m very sorry this PR having indicator bug.The calculation of percentage may be wrong… 🙈

[RupertAvery]

let me take a look at your branch. I need to catch up to your changes :)

[RupertAvery]

UPDATE: Oh nevermind, my exception catching was on for all exceptions, looks like the library is throwing exceptions internally when scanning what the file format is.

I'm trying out SharpCompress on my 7z files and I get this error:

SharpCompress.Common.ArchiveException: 'Failed to locate the Zip Header'

Would you know anything about it?

It happens here at :

var archive = ArchiveFactory.Open(stream)
```</del>

[RupertAvery]

It looks like you need to get the decompressed size of the current file:

                archive.EntryExtractionBegin += (sender, args) =>
                {
                    _notifier.Notify(PopstationEventEnum.DecompressStart, args.Item.Key);
                    _currentFileDecompressedSize = args.Item.Size;
                };

        private void ArchiveFileOnExtracting(object sender, CompressedBytesReadEventArgs e)
        {
            var percentage = ((double)e.CompressedBytesRead / (double)_currentFileDecompressedSize) * 100;
            _notifier.Notify(PopstationEventEnum.DecompressProgress, (int)percentage);
        }

However, this doesn't get it all the way to 100% all the time, but we can probably fix that somewhere:

Processing started: 21:57:47
Processing D:\roms\Breath of Fire III (USA).7z
Decompressing file Breath of Fire III (USA) (Track 1).bin - 99%
Decompressing file Breath of Fire III (USA) (Track 2).bin - 91%
Decompressing file Breath of Fire III (USA).cue - 100%
Multi-bin image was found!
Merging .bins...
Using Title 'Breath of Fire III'
Writing Disc 1 - 100%

We can do this just to bump it up to 100%.

             archive.EntryExtractionEnd += (sender, args) =>
                {
                    _notifier.Notify(PopstationEventEnum.DecompressProgress, 100);
                    _notifier.Notify(PopstationEventEnum.DecompressComplete, null);
                };

[RupertAvery]

I could be wrong about CompressedBytesRead. 7zip is weird in that you need to go through all previous files before you extract one file.

I don't see the progress update after extracting the first file. It doesn't do anything for a while, then jumps to 98% and finishes. Same with the last file.

Could be that CompressedBytesRead is the total extracted bytes. I'll look into it a bit more. It's nice to have a managed 7z unpacker though.

[RupertAvery]

I'm a bit confused. The property is "CompressedBytesRead", but it matches up more or less with the (uncompressed) Size.

Also the CompressedSize is 0.

At any rate, it's good enough for a progressbar.

[RupertAvery]

So I tested the percent calculation on the latest branch of SharpCompress (standalone, not using PSXPacakger) and funny enough, the percent reaches 100 all the time for all files.

Dunno if there are updates to the codebase that fix it, or I'm doing something wrong in PSXPackager (probably not, because I copied our code into a new project in SharpCompress to test it, and also using BoF3.

I also looked into the 7z unpacking part of SharpCompress, and yeah, there really is a delay when you unzip files further into the archive. It looks like there is an ongoing issue with it, as the 7zip format requires you to unpack/read the previous file in the archive before you unpack the one you want.

It's not a huge deal, but if you have many .bins in one archive, it will be much slower unpacking.

What happens is, suppose you have the files:

BoF3 (1).bin
BoF3 (2).bin
BoF3.cue

It will perform the following:

unpack BoF3 (1).bin -> output file
unpack BoF3 (1).bin in memory (throw away data) and count how many bytes have been "extracted", when EOF reached,
unpack BoF3 (2).bin -> output file
unpack BoF3 (1).bin in memory (throw away data) and count how many bytes have been "extracted", when EOF reached,
unpack BoF3 (2).bin in memory (throw away data) and count how many bytes have been "extracted", when EOF reached,
unpack BoF3.cue -> output file

This also happens in 7zip when you just want to extract BoF3.cue, but if you extact all the files, it can continue from the last position in the archive, I guess.

[takano32]

In fact, I did not expect to be treated so courteously! Thank you very much!

I see that my code was trying to calculate the progress of individual files from the size of a single compressed file, and that's why it wasn't working. I understand that calculating from the size of each individual file works fine.

I didn't expect the 7z expansion to be so complicated. Thank you for explaining the details. I learned a lot. I am not sure if I understood everything, though. 😅

Finally, this PR is now truly Ready for Preview! 🎉 Thank you very much. 🙏

[takano32]

If there seems to be a concern that using SharpCompress will degrade performance, perhaps it would be better to try using SevenZipSharp first and fall back if it is not available, instead of using SharpCompress only?

About 90% of the files I handle are in ZIP format and maybe 10% in 7z format. I am hopeful that replacing the library with SharpCompress will allow me to handle ZIP files in Linux and macOS, even if it will degrade performance.

But it is not my intention to be disparaging about performance.

[RupertAvery]

Don't worry about it. I was just mentioning it since I noticed the pause when extracting, and I thought you should know.

Eventually someone will come along and fix it. I tried, but I don't know enough about the LZMA format and the codebase.

I'm sure Linux/OSX users will be glad to have 7z support!

We can just make sure to inform users of the change and possible effects.