Add JPEG XL (.jxl) benchmark profile

[EwoutH]

Release v0.3.1 of JPEG XL was tagged yesterday, and this might be a good moment to add the first test profile.

The JPEG XL reference implementation has an encoding and decoding CLI apps, cjxl and djxl respectively. Syntax is quite simple:

• cjxl input.png output.jxl
• djxl input.jxl output.png

cjxl has the following speed (-s) settings:

    3|falcon| 4|cheetah| 5|hare| 6|wombat| 7|squirrel| 8|kitten| 9|tortoise
    Default: squirrel (7). Values are in order from faster to slower.

I would suggest testing 5, 7 and 9 (the fastest 3 isn't that interesting since there wouldn't be a great use case for it, and 5 and 7 are already pretty fast)

The decoding app, djxl, also has a benchmarking option which is multi-threaded. Just specifying djxl input.jxl (without an output) is enough. Could be interesting to test!

So in my view 4 test would be appropriate: Encoding with speed 5, 7 and 9 and decoding.

I found Windows-binaries here: https://encode.su/threads/3564-JXL-version-0-3-released?p=68594&viewfull=1#post68594

[michaellarabel]

Thanks, for now added some basic encode benchmarks with the new jpegxl test profile.

https://openbenchmarking.org/test/pts/jpegxl

[veluca93]

If you want my 2 cents, I'd add single-threaded options too (--num_threads=0).

Thanks a lot for adding JPEG XL to PTS!

[veluca93]

If you want my 2 cents, I'd add single-threaded options too (--num_threads=0).

Thanks a lot for adding JPEG XL to PTS!

I noticed that the current test profile tests lossless JPEG recompression - while this is an interesting usecase, I'd expect it not to be the principal one. Is there a possibility to edit the test profile? Should I just open a merge request?

[michaellarabel]

If you want my 2 cents, I'd add single-threaded options too (--num_threads=0). Thanks a lot for adding JPEG XL to PTS!

I noticed that the current test profile tests lossless JPEG recompression - while this is an interesting usecase, I'd expect it not to be the principal one. Is there a possibility to edit the test profile? Should I just open a merge request?

Yes you can use merge requests to stage the changes. (Ultimately it isn't pulled through GitHub as first goes through OpenBenchmarking.org versioning) but any improvements are welcome either through GitHub merge requests or simple patches.

[EwoutH]

Thanks Michael!

I noticed that the current test profile tests lossless JPEG recompression

Is -q 100 the default? That's a strange one, didn't know. I would support benchmarking lossy compression primarily!

[jonsneyers]

When the input is already lossy (a JPEG file), the default behavior of cjxl is to recompress it in a lossless way, to avoid creating any further artifacts.

When the input is a PNG file (or a PPM file), the default behavior of cjxl is to compress it in a lossy way, with a setting that should be visually lossless.

It would be interesting to test both code paths, since they are both interesting.

Speed 9 is probably not that interesting (it is very slow). Speeds 6, 7, 8 are in my opinion the most relevant ones.

[michaellarabel]

@jonsneyers yeah 9 was too slow so the test profile only goes up to 8.

So you are basically suggesting adding a PNG input in addition to the JPEG (non-XL) input?

[EwoutH]

So you are basically suggesting adding a PNG input in addition to the JPEG (non-XL) input?

If I'm correct, yes.

Here are two 100 MP .jxl images, encoded with cjxl 0.3.1 at speed 7, if you want to add a decoding test (djxl). One is lossless encoded and one at -q 75. Decoding speed is quite be important for browsers.

Image source: https://fujifilm-x.com/en-us/products/cameras/gfx100s/sample-images/

[jonsneyers]

PNG input is not strictly needed: with the option -j, cjxl will decode a JPEG input to pixels and treat it as if it was a PNG.

I've made some suggestions in a pull request (https://github.com/phoronix-test-suite/test-profiles/pull/187), feel free to add more tests like those 100 MP images.

I agree that decode speed is quite important for browsers. Probably for that, a smaller image (e.g. 1 MP) and single-threaded decoding matter more (a web page can have many images and a browser has other things to do than to decode images, so one thread per image is – I think – what tends to happen in practice).

[EwoutH]

It's interesting how much the speeds differ between the lossless compression (Modular mode) when JPEG input is given and the lossy (Var-DCT mode) with PNG input.

For the AMD Ryzen 9 5950X (source):

	JPG input	PNG input
Speed 5	85.7 MP/s	73.2 MP/s
Speed 7	85.5 MP/s	14.9 MP/s
Speed 8	36.8 MP/s	1.10 MP/s

[veluca93]

It's interesting how much the speeds differ between the lossless compression (Modular mode) when JPEG input is given and the lossy (Var-DCT mode) with PNG input.

For the AMD Ryzen 9 5950X (source):

JPG input PNG input Speed 5 85.7 MP/s 73.2 MP/s Speed 7 85.5 MP/s 14.9 MP/s Speed 8 36.8 MP/s 1.10 MP/s

Lossless JPEG recompression still uses VarDCT mode - the thing is, most heuristics can be skipped because you can't decide quantization levels or block sizes if your input is a JPEG and you want lossless, so it's going to be a lot faster.

[jonsneyers]

Modular mode (used for lossless compression of pixels, not of existing JPEGs) is even slower - both in encode and decode.

Part of the reason for slower speeds, especially in the encoder, is that not everything has been parallelized yet - and some encoder decisions cannot really be parallelized, because you want to globally optimize things for best compression.

A benchmark of decoding speed would be interesting, also because it should benefit more from multicore.

[michaellarabel]

A decode benchmark is now available as pts/jpexl-decode

https://openbenchmarking.org/test/pts/jpegxl-decode

Thanks