[Feature Request] Zstandard support (zip/7z/chd/zst)

[Sanaki]

With the recent additions of zstandard support to RomVault (zip, 7z, and chd), MAME (zip and chd, mamedev/mame#11827), and libchdr (chd, rtissera/libchdr#106), this is a prime time to add support to RetroArch for these formats as well. For anyone unaware, zstd provides high speed compression and decompression with respectable compression ratios (significantly improved over deflate on all counts, and competitive with lzma depending on compression level). It's also the standard compressor for Dolphin's RVZ format.

Support for chd will eventually be handled by updating the libchdr dependency, so that one's relatively simple. For zip, it exists in the official spec as method 93, though I'm uncertain how simple it is to hook that up to the zstd dependency added by libchdr's update. 7z support should be about the same as zip, though again, I'm uncertain how involved that would be.

Less straightforward is support for zst, which is a single-file compression format (similar to gz in that regard). It's unclear if there would be particular benefit to supporting bare zst files as single-file archives, or whether the difficulty of implementing that support would be prohibitive. I'd love to hear opinions on this point, beyond the obvious "why not?"

4122 is a similar issue on the surface, but is centered around zpaq suipport. I felt it prudent to open a new issue considering.

EDIT: Did some quick testing by adding cdzs to the Silent Bomber chd. First is without it, second is with. Significant improvement in read speed, which has advantages for mobile devices with limited I/O speed.

chdr-benchmark "Silent Bomber (USA).chd"

libchdr benchmark tool....
Read 356937984 bytes in 9.787469 seconds
Rate is 34.779429 MB/s

chdr-benchmark test.chd

libchdr benchmark tool....
Read 356937984 bytes in 7.134529 seconds
Rate is 47.711992 MB/s

I ran a further test on DVD type chds with pure zstd, and the difference is frankly shocking.

Standard DVD (Jak and Daxter US)
CHD size:     887,124,057 bytes
Ratio:        60.7%
Read 1460535296 bytes in 28.192692 seconds
Rate is 49.405534 MB/s

ZSTD-only DVD (Jak and Daxter US)
CHD size:     913,880,480 bytes
Ratio:        62.6%
Read 1460535296 bytes in 8.581942 seconds
Rate is 162.303008 MB/s

[InquisitiveCoder]

Just wanted to express my interest in the CHD support in particular, since the MiSTer FPGA project updated its libchd version two weeks ago. Zstd is great for MiSTer since it's the only (de)compression algorithm fast enough to use x8 CD speed, so if RetroArch supported zstd CHDs I could use one set of CHDs for both platforms.

[saulfabregwiivc]

The new CHD ZStandard compression must be added to each emulator which supports/loads CHD, not RetroArch. RetroArch is just a front-end (correct me if i'm wrong).

[Sanaki]

Both are necessary for different reasons. RetroArch support is needed for achievements to work.

[LodanZark]

The current Retroarch nightly build was able to run CHD-ZSTD with the upstream flycast and ppsspp cores (might be the only cores to support that format so far).

[Immersion95]

I think the frontend support is needed in RA when handling with roms inside archives (SNES, N64, etc...). Roms as archive should be handled by the core (CHD, MAME's zip).

[hizzlekizzle]

I believe the holdup on this is that it needs to basically happen all at once in RetroArch and the various libretro cores due to the way statically linked platforms work.