unknownbrackets / maxcso

Fast cso compressor
ISC License
403 stars 24 forks source link

Allow larger block sizes than 256 MB #89

Open ElektroStudios opened 3 months ago

ElektroStudios commented 3 months ago

This is a feature request and mostly a real world requirement in these days:

Please allow larger block sizes than 256 MB, up to 8 GB.

We are in the era of modern technology and competent PCs, where any advanced user should be able to specify a larger block size in order to (try)reduce the size of generated CSO file.

I have zero issues running CSO files compressed with maxcso using the maximum allowed 256 MB block size with zopfli, loading the CSO files in PCSX2 and PPSSPP. And my computer is from year 2017, it is not and it was not a high-end PC in any way.

Some PS2 ISO files from 'REDUMP' like "God of War (Europe, Australia) (En,Fr,De,Es,It).iso" are very huge, in this case 7,46 GB, and nothing -except 7zip- compares to the file size reduction that I can achieve using WinRAR (RAR5 format) with a dictionary size of 2GB, which generates a RAR file of 5,33 GB compared to the CSO file of 6,26 GB that maxcso generates using zopfli with 256 MB block size (by the way, it is the same resulting rounded file size as using zopfli with 128 MB in this case, for this ISO).

256 MB block size falls extremely short. Just please consider to allow the usage of larger block sizes (starting with 512MB, 1024MB, 2048MB and so on) for those who just want to test the compression results and who understand and accept the possible emulation issues on some very old computers and portable devices when increasing block size.

As an example, DolphinTool's CLI tool from Dolphin Emulator for GameCube and Wii, which serves to compress game files to RVZ format, it is even worse by warning all the time about emulation issues just for using 4MB block size and above (I never had any emulation issue on thousands of games GameCube and Wii games that I've tried with very larger block sizes), however it allows any larger block size assignation and the operation is completed as normally, it just warns about it but it allows the user to use whatever block size his memory slots can achieve...

Final note: I really don't understand how block size / dictionary size from WinRAR works in compression, I just proposed 8 GB as a limit since it is around the maximum file size I've seen for biggest PS2 ISO files. I know that a dictionary size of 6GB in WinRAR can profit / can make smaller file size when compressing a ~7 GB PS2 ISO file, but I'm not sure about 8GB dictionary size in this case, and in any case I'm not sure whether RAR's dictionary size is in fact the same as what we refer by the term 'block size' for other compression algorithms, but I think it should be very similar just different names.

UltimateNova1203 commented 2 months ago

Something to keep in mind is real hardware PSP compatibility, wouldn't be the first time (in general, not specifically this tool) hardware compatibility was broken from changes for emulation users.

I'm not sure if increasing the dictionary size would negatively affect hardware compatibility or not, or if there is an upper limit to it.