Color distance should be calculated in a uniform color space

[AbelVM]

Found this function in the cpp_rework branch that calculates the difference between two colors as the euclidean distance of the RGB coordinates. That's far from optimal, as it only makes sense when only one of the color channels changes while the other two stay the same.

The difference between colors can be expressed as the euclidean distance only when they are expressed in a perceptual uniform color space, being the most common CIELAB or CIELUV. The most common metric for color distance is ΔE* 2000 that adds a lot of corrections to the original distance formula and that renders the algorithm a bit tricky. So, as the target here is performance, I'd go with CIELUV and then the Euclidean distance of the (L, u, v*) coordinates. OpenCV offers a RGB --> CIELUV conversion function so it should be easy to implement.

On the other hand, if this function is used on a per-pixel basis, it may make sense to convert the whole frame to CIELUV using OpenCV cvtColor function

[akai-katto]

Thanks for taking a look at my work in progress branch: today's a perfect day to try it out since I'm experimenting with how to normalize the input data. I wasn't aware this existed despite working on cpp for a while.

[akai-katto]

I worked on this for the evening - it's looking very promising.

There's some issues with gradients, but overall if I can see if I can get it to work, this might be the future for the computing mechanism, if all goes as planned.

[AbelVM]

This might also apply to the apply_noise function, indeed.

RGB is just a color model (good for maths, bad for humans). Check this green->blue (hue) gradient in sRGB

And same gradient in HSLuv (HSL-like CIELUV)

[akai-katto]

I've done extensive testing with the function and it works perfect on paper, but there's a disconnect between waifu2x-ncnn-vulkan upscaled images and this. I'm not sure what's causing the differential in behavior: where as mean squared error over RGB space scaled well, it seems mean squared error over LUM space has blocking issues, especially in a specific color set.

I'm not giving up on it entirely + still playing around with it, since in all other scenario's its working well, just the niche case of certain colors having issues once upscaled (i.e looking good in lower resolution, but upscaled looks entirely different).

Great find, i'm continuing to fiddle with it trying to get it to work.

[AbelVM]

Do you have any samples that you can share of that weird behavior? And some context on the use of the square function would be great.

My suggestion was to use a uniform color space to evaluate color distance and, maybe, adding noise. Anywhere out that scope might mess with the models if they expect RGB channels.

On the other hand, you might want to check CIELAB if CIELUV gives you weird results

[akai-katto]

Found a fix to it, I modified the original code I found to heavily punish the luminosity layer when preforming comparisons. The luminosity later needed to be treated non-linearly, since shades of colors weren't being punished enough (had severe consequences within waifu2x). What ended up happening before was flickering due to mismatches with predicted block matches, but were really different shades after being ran through waifu2x.

Hoping to post some research updates on it today -> do more testing, but finally got this to work after a week of toying around with it.

[AbelVM]

Yep! Human perception of color is logarithmic, not linear! And, using CIELUV you might want to use two different color distance metrics:

• Lightness difference (plain delta): L_b - L_a
• Chromaticity distance (euclidiean for constant lightness): sqrt((u_b - u_a)²+ (v_b - v_a)²)

[akai-katto]

Playing around with other color spaces as time goes on. RGB seems to be a poor space to use, so still looking at other alternatives.

[akai-katto]

oof, implemented the wrong lab conversion code.

[AbelVM]

Hmmm... some questions

Dandere2x uses OpenCV actually... so, why don't you just use the color conversion functions there instead of coding your own conversions twice at MSE_FUNCTIONS::RGB2LAB and get_lab_from_rgb? That's not much DRY, and the algorithms to translate RGB to LAB are different in each function, so that will yield inconsistent results.

Python style, converting one color value with OpenCV might be done just like in just one line (ref), in cpp it should be not so different:

import cv2
import numpy as np
bgr = [40, 158, 16]
lab = cv2.cvtColor( np.uint8([[bgr]] ), cv2.COLOR_BGR2LAB)[0][0]
print(lab)  #[145  71 177]

(but using cv instead of cv2, using the enumerator that fits your conversion and so on)

The same cvtColor function may be used to translate a full frame, no need to make a loop there (convert_to_lab)

[AbelVM]

Any update on this issue? Looks like latest release is still using RGB distance, without taking into account lightness and chromaticity as independent dimensions, like CIELUV would

• Lightness difference (plain delta): L_b - L_a
• Chromaticity distance (euclidiean for constant lightness): sqrt((u_b - u_a)²+ (v_b - v_a)²)

[akai-katto]

I haven't had the chance to implement the option for different "distance" metrics. If time permits, this can be another c++ feature, but putting this off for the time being to get some other dandere2x issues sorted out.