Open Dimava opened 2 years ago
Yes, I think this point could easily be missed.
In my testing, combining QOI with ZLib compression (QOZ) produces file sizes significantly smaller (roughly 10-25%) than when using PNG format. Of course this gives up most (but not all) of QOI's speed advantages, but QOZ still retains a significant (2-5 times) speed advantage when compressing.
note that zlib is no longer a good choice for compression. QOI+ZSTD or QOI+LZ4 are going to give much better compression vs speed tradeoffs.
I chose ZLib because its libraries are so widely available and for simplicity of adoption.
To me, that feels like a weird argument for an image format that's less than 2 months old.
I agree with @oscardssmith . Limiting recommendations for backwards compatibility when no existing support exists makes no sense
Just a random thought - does re-compressing QOI with QOI yield anything interesting?
no. QOI is very specifically designed for 3 or 4 channel images. The result after QOI compression isn't that.
no. QOI is very specifically designed for 3 or 4 channel images. The result after QOI compression isn't that.
Yeah, sure, I know that. It's just sometimes compression with one algorithm creates certain patterns that can be further compressed with another. bzip2-compressed mysql dumps can be compressed further 10% by gzip, even though the latter is "less sophisticated". This sort of thing.
So for a new compression algorithm it's worth checking if applying it recursively may show some interesting results. It's exceedingly rare, but when it happens it's spectacular.
I did a quick test just now and QOI can't compress further its own output. 400 x 400 photo crop went from 468K to 204K to 269K. So it's a no.
I mean, that's completely unsurprising though. Why would variable-length data chunks of QOI be able to be compressed by opcodes which describe difference-in-RGB movements? lol
I chose ZLib because its libraries are so widely available and for simplicity of adoption.
LZ4 and ZSTD are widely available too, the only likely places that they aren't available is for micro controllers or other hardware that aren't physically capable of running the algorithms. On Debian/Ubuntu adding basic support is the same for all three: Installing liblz4-dev/libzstd-dev/zlib1g-dev from the standard repo, including lz4.h/zstd.h/zlib.h, and the half a dozen lines it takes to call the respective simple encoder/decoder.
Yeah, sure, I know that. It's just sometimes compression with one algorithm creates certain patterns that can be further compressed with another.
A smart QOI-recompressor would at least take advantage of QOI being a byte-aligned format. The tags might be ANS-coded, the RGBA elements may be grouped for further processing, etc. The problem is that QOI output being byte aligned still allows for entropy coders to get good results so a QOI-recompressor would have to be excessively smart to compete.
I chose ZLib because its libraries are so widely available and for simplicity of adoption.
LZ4 and ZSTD are extremely easy to get working in your program. Yesterday I tried using both of them for my own game files package system and it works, just few files .c and .h files from their repo, call one function to compress, one to decompress... I recommend trying it out.
dotnet has had a Brotli implementation since core 2.1 (but no zstd) and qoi unsurprisingly works well with Brotli too.
With compression, qoi looks to be better than PNG, perhaps there could be a Compression byte in the header, 0 for none and other values assigned to presently-known algorithms. However, this would significantly complicate interpretation of interchanged .qoi files - i.e. files from diverse sources.
Having gzip or brotili or whatever other compression format wrap a qoi file is good enough; there is no reason to make those external compressions a part of the qoi format.
It can be seen that in many cases, QOI+DEFLATE compresses better than PNG, but sometimes it isn't a better compression. PNG alone is usually better than QOI alone, though. (The below is a list of the contents of the ZIP archive with the test files)
Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2021-12-21 04:47:44 D.... 0 0 qoi_test_images
2021-12-21 04:47:44 ..... 349827 333282 qoi_test_images/dice.png
2021-12-21 04:47:40 ..... 519653 350872 qoi_test_images/dice.qoi
2021-11-21 10:18:50 ..... 593463 593449 qoi_test_images/kodim10.png
2021-12-21 04:40:01 ..... 652383 522804 qoi_test_images/kodim10.qoi
2021-11-21 10:19:32 ..... 557596 557596 qoi_test_images/kodim23.png
2021-12-21 04:41:07 ..... 675251 544532 qoi_test_images/kodim23.qoi
2021-12-08 16:06:13 ..... 16605 16526 qoi_test_images/qoi_logo.png
2021-12-21 04:44:14 ..... 16488 8107 qoi_test_images/qoi_logo.qoi
2021-12-21 04:40:17 ..... 14227 13619 qoi_test_images/testcard.png
2021-12-19 15:18:05 ..... 21857 7402 qoi_test_images/testcard.qoi
2021-12-21 04:47:25 ..... 18371 17692 qoi_test_images/testcard_rgba.png
2021-12-19 15:21:21 ..... 24167 9453 qoi_test_images/testcard_rgba.qoi
2016-07-19 08:19:40 ..... 1344960 1328424 qoi_test_images/wikipedia_008.png
2021-12-21 04:42:27 ..... 1521134 1343930 qoi_test_images/wikipedia_008.qoi
------------------- ----- ------------ ------------ ------------------------
6325982 5647688 14 files, 1 folders
Yes, I think this point could easily be missed.
In my testing, combining QOI with ZLib compression (QOZ) produces file sizes significantly smaller (roughly 10-25%) than when using PNG format. Of course this gives up most (but not all) of QOI's speed advantages, but QOZ still retains a significant (2-5 times) speed advantage when compressing.
Yes I confirm. We actually made our internal format based on this: qoi + lzma. You can try it out at https://github.com/sumo-apps/voiconv .. The benefit of "qoi" before lzma is at least to decrease (significally) the size of data going to heavier compression.
Oh my, this thread is full of spam, discussing irelevant off-topic stuff. Including semi-personal attacks, now including mine.
Getting back to the original topic of practical further compression: There are use cases for this, imagine (ha ha) having a game with thousands of images. You want to be able to quickly decompress random files but for downloading that game, you can keep it all relatively small. Or decompress from qoi-set.zstd into memory and decode the qoi lazily.
And then, every dev using qoi as a library can just wrap it in zstd or similar by self, no need to discuss extending the format. There might be even more microcontroller friendly decompressors that might be combined with qoi for tiny flash+memory footprints. Now, benchmarking that kind of stuff would be helpful and bringing this topic forward!
Somebody wants to benchmark a test image set with a selection of some common/uncommon compressors? (Test both single qoi and a set of qoi per compression run!) Then we could add this to the readme!
I did some test of 'entropy coding' for qoi (compressing with regular compressors). Used subset from benchmark images and default compression levels (except zstd, which was set to 12, to somewhat match gzip level 6. Scripts used to make this test, in case someone wanted to test it themself are also attached.
That's the result:
20737618 qoi.bmp
7035249 qoi
6630663 qoi.lz4
5850003 qoi.gz-H
5721098 png
5476417 qoi.zst
5471989 qoi.gz
5113443 qoi.lzma
5102454 qoi.bz2
Here are some more elaborate results (with times and calculated speeds):
# sorted by ratio
# method ctime cspeed dtime dspeed compr. ratio
qoi.bz2 2.762 7.5082 1.491 13.9085 5102454 0.246048
qoi.lbz2 2.124 9.7634 1.122 18.4827 5102454 0.246048
qoi.lzma 5.934 3.4947 1.534 13.5187 5113443 0.246578
qoi.gz 1.410 14.7075 0.543 38.1908 5471989 0.263868
qoi.zst 2.061 10.0619 0.491 42.2355 5476417 0.264081
png 10.308 2.0118 2.361 8.7834 5721098 0.275880
qoi.gz-H 0.859 24.1416 0.549 37.7734 5850003 0.282096
qoi.lz4 0.533 38.9074 0.409 50.7032 6630663 0.319741
qoi 0.437 47.4545 0.332 62.4627 7035249 0.339251
# sorted by ctime
# method ctime cspeed dtime dspeed compr. ratio
qoi 0.437 47.4545 0.332 62.4627 7035249 0.339251
qoi.lz4 0.533 38.9074 0.409 50.7032 6630663 0.319741
qoi.gz-H 0.859 24.1416 0.549 37.7734 5850003 0.282096
qoi.gz 1.410 14.7075 0.543 38.1908 5471989 0.263868
qoi.zst 2.061 10.0619 0.491 42.2355 5476417 0.264081
qoi.lbz2 2.124 9.7635 1.122 18.4827 5102454 0.246048
qoi.bz2 2.762 7.5082 1.491 13.9085 5102454 0.246048
qoi.lzma 5.934 3.4947 1.534 13.5187 5113443 0.246578
png 10.308 2.0118 2.361 8.7834 5721098 0.275880
# sorted by dtime
# method ctime cspeed dtime dspeed compr. ratio
qoi 0.437 47.4545 0.332 62.4627 7035249 0.339251
qoi.lz4 0.533 38.9074 0.409 50.7032 6630663 0.319741
qoi.zst 2.061 10.0619 0.491 42.2355 5476417 0.264081
qoi.gz 1.410 14.7075 0.543 38.1908 5471989 0.263868
qoi.gz-H 0.859 24.1416 0.549 37.7734 5850003 0.282096
qoi.lbz2 2.124 9.7635 1.122 18.4827 5102454 0.246048
qoi.bz2 2.762 7.5082 1.491 13.9085 5102454 0.246048
qoi.lzma 5.934 3.4947 1.534 13.5187 5113443 0.246578
png 10.308 2.0118 2.361 8.7834 5721098 0.275880
As can be seen lz4 is inefficient in this application, which becomes clear when one realises that lz4 does not use any entropy coding, and the entropy coding, even pure prefix code like huffman from deflate, is what makes a difference in this contest. Lzma is very tight but still loses to bzip2, both in terms of compression and speed; zstd turns out to be worse and slower than gzip (only decompression is slightly faster but it doesn't make up to it.
All in all, the best compressor (by ration) is bzip2, and by speed gzip/deflate, which beats png both, with compression (4%-5%) and with speed (~7 times).
Unlike most of other image formats which can not be compressed further, QOI can be considerably compressed with general compression algorithms Unlike other image formats which will generate highly binary different files for partially same images, QOI generates alike files, alloving to compress variant sets with insane compression (I've got 11MB .qoi.7z out of 4x 12MB pngs)
Please write the information about high compression rates for both single images and image sets in the readmes