Very noticeable color shift

[ScopeBurst]

Spotted by a user in AV1 Discord when encoding one of the images with generated noise Source: song.png Full resolution encoded jxl -d 1 -s 7 song.d1s7.jxl.png

Cropped 32x32 source:

Encoding with -s 9 helps somewhat, but the color shift is still quite noticeable

[jonsneyers]

This is an interesting phenomenon; it would be good to understand better what is causing it. @jyrkialakuijala might have insights here?

[jyrkialakuijala]

Obviously something is wrong. It could be averaging the gamma compressed values or chroma-from-luma computations (for the blue-yellow chroma).

[eugenesvk]

Is there any indication re. what kind of input this bug might affect (the original pic with generated noise is rather artificial) just to have some understanding re. potential risks? Also, given that color shift is a rather substantial quality degradation, wouldn't it be better to try to fix it before v1.0? Thank you

[jonsneyers]

I haven't seen any non-artificial image on which this happens.

At effort 3 and 4 it does not happen; only at effort 5 does it start to happen. That's also the effort where chroma-from-luma starts to be used, so it could indeed have something to do with that, as @jyrkialakuijala suggested.

Probably worth investigating what exactly is happening here.

[eugenesvk]

I haven't seen any non-artificial image on which this happens.

Thanks, that's reassuring; I was rather worried this bug might creep into some large batch job unnoticed.

only at effort 5 does it start to happen

but those are the best efforts ;) and include the default of 7

[jendalinda]

It doesn't seem to be that bad at effort 9.

[jojje]

I notice this as well. Especially with blue eyes. They start quickly shifting towards green/brown. I can spot it with the naked eye at distance 0.2 already. And at that quality, the image is way larger than jpeg at quality 95 and chroma subsampling disabled.

Initially I thought this was a regression in the latest release, but it's present also in v0.8.2. I can accept blockiness and general quantization defects, but chroma shift is unfortunately one artifact that is unacceptable to me, preventing me from switching over to this format. So I'll have to stay with jpeg q95 without chroma subsampling it seems :(

Attached are examples showing the shift. Watch as the right eye (on the left) moves towards green the higher the distance value. Already at 0.2 the shift is noticeable. At 0.5 definitely, and at 1.0 you'd have to be blind not to see it.

cjxl-chroma-shift.zip

the png is the reference image, and the encoding was done using cjxl v0.10.3 4a3b22d with cjxl -e 6 -d <distance> eyes.png eyes-d-<distance>.jxl

[jyrkialakuijala]

Do you observe the color shift by zooming or in usual viewing conditions?

If zooming how much do you zoom to make it noticeable?

I'm considering adding a flag or possibly two flags that would remove, reduce, or modulate the psychovisual color modeling when there are needs for extreme zooming or/and the greatest color fidelity.

[jojje]

Hi Jyrki. I only zoom when there are smallish images. Typically have my image viewer (irfanview) set to full-screen view, which naturally zooms (upscales) images smaller than the native screen resolution.

However, the color shifts like the one I posted, I can spot even when not zoomed. Something just seemed off on some images, so I then zoomed a bit and saw it was indeed the colors that were wrong.

But yes, I get your point. I posted very tiny examples, and it is understandable you thought I had zoomed those tremendously to spot the issue. When I compare with larger images, like high-res eyes, the encoder doesn't seem to shift the colors noticeably, so I would assume it has to do with small details, where the encoder thinks it can get away with playing with colors the same way it's allowed to play with "macro-blocks"; just another degree of freedom for it to optimize.

EDIT: The flag you mentioned, to improve color fidelity would be most welcome. And as for your question about special viewing conditions. It depends on what you mean special. Normal viewing condition in a typical Finnish-like home with typical northern lighting condition (not LA blazing sunlight), and a Viewsonic monitor able to show colors properly.

[jonsneyers]

On the d0.2 image, I cannot see a color shift. On the d0.5 image I can see a small amount of color loss, on the d1 image a larger amount. I can only clearly see it if I zoom in quite a bit though.

When viewing the image at my native screen resolution, meaning this image is about 1 cm wide, I cannot see a difference between the original and the d1 image when viewing from a normal viewing distance (about an arms length). When I zoom it 10x to about 10 cm wide, the difference is very obvious. This makes sense, considering the distribution of S cones in the fovea.

[jojje]

Why colors matter more than "jaggies" (structural similarity) Jon, is that the latter can be alleviated rather well with today's current image "restoration" tools (pixel dream machines) like Gigapixel AI and the like, and they'll likely only get better in the future. This has already allowed me to sufficiently salvage very old pictures of shoddy quality I thought were meant for the recycle bin. Colors however, that's something which wreaks havoc with such tools, since they'll amplify the incorrect colors, making the image diverge even more from the ground truth. So even if you can't spot the color shifts, once a small image has been processed in such a manner and increased in resolution, I bet even you would be able to see it on your 4K monitor at native resolution ;)

Edit: And for comparison. Save the PNG to "old" JPEG at q95 with no chroma subsampling and compare. It retains the colors, but is of course a lot more macro-blocky than XL, which is to be expected.

[jonsneyers]

Any information that is lost is of course lost, I don't think that loss of luma can inherently be 'alleviated' better than loss of chroma, though it's possible that current restoration tools are mostly aimed at repairing luma artifacts.

I think we could expose an encode option to adjust the balance between luma and chroma, and/or to bump up the fidelity of the blue-yellow chroma component specifically. For images that will be upsampled, or displayed with "big pixels", it would make sense to adjust that balance.

If the use case is to encode images at low resolution and then upsample them 10x, then that's quite a bit outside of the viewing condition assumptions the libjxl encoder is making. Lossy compression is all about modeling what the human visual system can and cannot see; one of the things libjxl is exploiting is that our S-cones have a lower spatial resolution than our L and M cones, which means that the high frequencies in the blue-yellow chroma component (the B of XYB) will be compressed more aggressively than in the red-green chroma component (the X of XYB) or in luma (the Y of XYB). That's also why you don't get a color shift if the image is larger to begin with (and the pixel area of the eyes is larger), but you do get one it if it's small like that (here it's basically a 2px wide blue ring around a black pupil, so pretty high frequency info in the B component).

Maybe the current balance is not quite correct, if you can see the color shift without zooming in. The aim of d1 is to be at the just noticeable difference point, i.e. the difference is not visible for half of the observers with normal color vision (while it is visible for the other half), when viewed at native resolution, at a typical viewing distance, on a typical display.

[jyrkialakuijala]

I have found another mitigation. I consider it will fix about 15 % of this issue. I'll send the PR in within three days.

Basically I look for blue colored pixels that are above the red and green values (so that red/green don't mask the blue), and if a 8x8 block has about 32 such pixels then it gets a boost in adaptive quantization, i.e., some more bits. As a downside it will slow down the encoding a tiny bit.

[jojje]

Thanks Jon for making it clear the design objectives. Always good to have those front and center in order to adjust expectations. And to you as well Jyrki for being curious about what tweaks could be made to lessen the specific observed phenomenon.

[jyrkialakuijala]

--save_decompressed is temporarily broken in benchmark, so I haven't looked how this actually looks yet

for the blue eyes test it seems to make a minor improvement

[jyrkialakuijala]

related to https://github.com/libjxl/libjxl/issues/414#issuecomment-1267292882: disabling chroma-from-luma didn't seem to make a positive (or negative) change on the blue eyes

~4 years ago or so we had chroma-from-luma oddities, but seems like we have been able to recover from them solidly

[RubenKelevra]

@Galaxy4594 made me aware if this bug report, which fits my observed issue better. So let's quote my posts here again:

I can confirm some issues with 0.10.3 (4a3b22d2). I noticed that artificial noise in CGI/painted images is not properly processed, until you go up to ~ -q 96.

Which leads to a visible color shift of large whole areas, as they become brighter:

original: jxl with -q90 --progressive:

---

Not sure if helpful, but the issue is mainly more red and less saturation, as well as slightly lower value (average on a color plane over 40x40 px):

original:

jxl with -q90 --progressive:

Originally posted in https://github.com/libjxl/libjxl/issues/3530#issuecomment-2276551935