zstd support

[Jenmet-cy]

Hi. Are there plans to include zstd support in maxcso? I've read that it can compress better than lzo without beign as slow to compress as deflate.

[unknownbrackets]

maxcso can include CSO support, but note that does not automatically mean software that uses CSO files (such as PSP/PS2 emulators, custom firmware, firmware plugins, etc.) will support it. maxcso was originally designed, like pngcrush, to create smaller CSO files rather than invent a new format (just like pngcrush works for widely supported PNGs.)

The PNG analogy is actually good, because PNG also uses zlib and could use zstd. Even if pngcrush added some zstd mode, it wouldn't make web browsers able to display those PNGs.

With all that said, I may investigate. There's some easy ways to save a few more bytes as well, which would be fast to decompress. This would theoretically be a "CSOv3" format, and would be easy to support in PPSSPP since we already use zstd there.

Also note that maxcso is intentionally slow to compress. It's actually compressing the file about six times, typically. It does this to find the smallest possible output - that's why its name includes "max". You can use --fast to only use one trial, which will be massively faster, but produce larger files.

zstd also has tuning parameters, as does lzma - most compression algorithms do. There are generally a few options to create a small file:

• Pick a typically-best set of parameters, and accept that you'll lose by a couple percent on some files. A lot of software does this.
• Try a set of typically-best sets of parameters, and use the smallest one. A smaller file may exist, but the likelihood of a couple percent is greatly reduced. maxcso does this.
• Trial multiple parameters, and estimate based on their results other parameters that may give better results. This can be pretty slow. I believe pngwolf and kzip do this, for example.
• Compute more systematically which parameters are best, instead of relying on guesses. This can be /very/ slow. zopfli does this (included in maxcso, but disable by default because it's really very slow - much slower than compressing the file only 6 times over.)

-[Unknown]

[crashGG]

The biggest advantage of zstd should be its decompression speed and system overhead, which is close to the speed of copying. Higher speed means lower cpu overhead. It has important value in some low-end devices, such as fanless cpu, portable devices,Can reduce stuttering and increase battery life.Since ppsspp already supports zstd, why not try to let cso also support it ? :)

[malvarenga123]

The biggest advantage of zstd should be its decompression speed and system overhead, which is close to the speed of copying. Higher speed means lower cpu overhead. It has important value in some low-end devices, such as fanless cpu, portable devices,Can reduce stuttering and increase battery life.Since ppsspp already supports zstd, why not try to let cso also support it ? :)

That's all achieved with ZSO. LZ4 is faster than ZSTD.

[crashGG]

The biggest advantage of zstd should be its decompression speed and system overhead, which is close to the speed of copying. Higher speed means lower cpu overhead. It has important value in some low-end devices, such as fanless cpu, portable devices,Can reduce stuttering and increase battery life.Since ppsspp already supports zstd, why not try to let cso also support it ? :)

That's all achieved with ZSO. LZ4 is faster than ZSTD.

Although lz4 is faster than ZSTD, the compression rate of zstd is between deflate and lzma.zstd is a balance of efficiency and effectiveness.

[malvarenga123]

Your post from 2022 only mentions decompression speed and that is better with LZ4 than Zstd. Not only that, LZ4 is also between deflate and LZMA, according to https://mcmilk.de/projects/7-Zip-zstd/

[mirh]

Putting aside that LZMA2 decompression speed got some noticeable bump since Igor started to ship assembly optimizations some years ago, Zstd decompression speed is 4-5 times better than lzma while still staying fairly close to its compression ratio. Lz4 sure thing is like 100 times faster (and I believe it's probably the best of the "super dead fast" algorithms) but even the HC variant can barely get you twice as big files. And I believe you mixed it up with bzip in reading the graph.

[malvarenga123]

And I believe you mixed it up with bzip in reading the graph.

That's true, I stand corrected.

LZMA2 decompression speed got some noticeable bump since Igor started to ship assembly optimizations some years ago

I don't think it will make much difference to other platforms (not Intel or AMD) like is the case with the PSP, PS2 or PS3.

[mirh]

Indeed, I'm not even sure if x86 got all of them lol. And for as much as I would actually normally consider zstd "the barest acceptable minimum" as far as compression goes (and yes, with LZMA usually being my go-to flavor) I understand that if this is also needed on super limited platforms it may instead be already be too much. Though it should be noted zstd has a "fast" tradeoff too, where it gets you to just half of the decompression speed of lz4 but with a still slightly better compression ratio