Don't get weights for distances the algorithm can't handle (>2.0)

[Jules-A]

Fixes https://github.com/Artoriuz/glsl-chroma-from-luma-prediction/issues/5 and other artifacts like borders disappearing ect. , https://github.com/Artoriuz/glsl-chroma-from-luma-prediction/issues/10 (I think the only reason was because it wasn't respecting it).

Downside is scaling of text ect. doesn't look as great, you'd need to consider switching to an algorithm that can handle a radius >2.0 (I've had pretty good results with Garamond), even Hamming seems better if you need speed and sharpness loss is acceptable.

[Jules-A]

Ofc I forgot about the debug file (nvm), anyway the PR was mainly so it could be tested easier as it's probably not the best solution (I feel like changing algorithm would be better).

[Artoriuz]

You do realise the d2 variable stands for the square of distance, right? Distance goes from 0 to 2, distance² goes from 0 to 4, which is exactly why the line above this limits d2 to 4.0.

You're doing the equivalent of limiting distance to ~1.4142, which creates an early cut-off in the kernel.

This is how it currently works:

This is what you're proposing:

[Jules-A]

You do realise the d2 variable stands for the square of distance, right? Distance goes from 0 to 2, distance² goes from 0 to 4, which is exactly why the line above this limits d2 to 4.0.

You're doing the equivalent of limiting distance to ~1.4142, which create an early cut-off in the kernel.

I'm not sure why but for some reason it fixes all the artifacts I was seeing, can you explain why pow is used? I can't seem to figure out why even limiting it to something like 2.5 wasn't enough to fix the artifacts when that should be well under 2.

[Artoriuz]

can you explain why pow is used?

If you mean for d2, it's because the polynomial approximation of lanczos from fsr is defined for d2 instead of d. It's done like this for performance reasons.

[Artoriuz]

And I wouldn't mind changing this function to something else, I've simply settled with it because it offers the best performance x quality compromise at this radius (remember cfl only has access to the 12 nearest pixels, it can't use larger kernels for spatial resampling).

[Jules-A]

And I wouldn't mind changing this function to something else, I've simply settled with it because it offers the best performance x quality compromise at this radius (remember cfl only has access to the 12 nearest pixels, it can't use larger kernels for spatial resampling).

@Artoriuz Hmm... it looks like the problem is elsewhere, is there an easy way to check all the distances for correctness? I tried a whole heap of algorithms like this: https://pastebin.com/raw/QXTPLL8g (code taken from alt-scale for convince).

but at distance/radius > 1.5 there seems to be issues with all. Ironically most seem to be going in the opposite direction and making things too dark with the exception of bicubic but quality seems to suffer still.

[Jules-A]

This is master:

Hamming with radius/max distance of 1.5:

Profile=fast without luma scaling for reference:

Hamming is only ever so slightly slower. (6100 vs 5900).

[Artoriuz]

The negative lobe between 1 and 2 will always make "things go in the opposite direction", that's just the filter overshooting.

[Artoriuz]

CfL master:

CfL without the negative lobe:

Notice how all chromatic transitions are much blurrier in the second screenshot.

[Jules-A]

The negative lobe between 1 and 2 will always make "things go in the opposite direction", that's just the filter overshooting.

No, I meant, opposite to how CFL is currently severely over-saturating the image, they are under-saturating instead. In the case of master, I'm not convinced it's just overshooting, there's tonnes of artifacts like colour bleeding, excessive ringing, clipping and many other artifacts that were all solved by (effectively) limiting the distance to 1.5.

CfL without the negative lobe:

I don't understand what exact settings you mean by doing that.

Notice how all chromatic transitions are much blurrier in the second screenshot.

Yeah, but it's not very noticeable, whereas I can spot the discolouring in the eyes leaning all the way back in my chair immediately.

[Artoriuz]

This is the reference picture with 4:4:4 chroma, the eyes are not supposed to be "discoloured". There's not much to infer from luma in the region so the spatial filter needs to do most of the heavy lifting. If you remove the negative lobe you'll obviously remove the overshoots and the ringing, but it'll also make everything considerably blurrier which also means losing the fine detail in the eyes.

[Jules-A]

I don't know about limiting FSR to 1.5 but with Hamming I decided to try out image comparison metrics and I don't know if it's just your test image not great at showing worst case scenarios or if I did something wrong since the results seem so close?:

Including LUMA scaling:

FSR: 16.7445 (0.000255504)
HAMMING: 16.7498 (0.000255586)

PSNR
FSR: 64.261
HAMMING: 64.2573

DSSIM
FSR: 0.000178577
HAMMING: 0.000178505

[Artoriuz]

As for:

is there an easy way to check all the distances for correctness?

These are the 12 pixels in a given shader pass:

// 12-tap kernel.
//    b c
//  e f g h
//  i j k l
//    n o

And this is where the distances are defined, all against the top-left reference f:

  wd[0]  = comp_wd(vec2( 0.0,-1.0) - pp);
  wd[1]  = comp_wd(vec2( 1.0,-1.0) - pp);
  wd[2]  = comp_wd(vec2(-1.0, 0.0) - pp);
  wd[3]  = comp_wd(vec2( 0.0, 0.0) - pp);
  wd[4]  = comp_wd(vec2( 1.0, 0.0) - pp);
  wd[5]  = comp_wd(vec2( 2.0, 0.0) - pp);
  wd[6]  = comp_wd(vec2(-1.0, 1.0) - pp);
  wd[7]  = comp_wd(vec2( 0.0, 1.0) - pp);
  wd[8]  = comp_wd(vec2( 1.0, 1.0) - pp);
  wd[9]  = comp_wd(vec2( 2.0, 1.0) - pp);
  wd[10] = comp_wd(vec2( 0.0, 2.0) - pp);
  wd[11] = comp_wd(vec2( 1.0, 2.0) - pp);

Turning off both regressions outputs this:

Which isn't that different from asking mpv to upscale chroma using a sharp polar bcspline + AR:

The only obvious difference is the checkerboard pattern, but I'm not sure if you're talking about this.

[Artoriuz]

I don't know about limiting FSR to 1.5 but with Hamming I decided to try out image comparison metrics and I don't know if it's just your test image not great at showing worst case scenarios or if I did something wrong since the results seem so close?:

I don't know what you're doing either but FSR has always scored rather poorly. Lanczos already scores higher than it on most distortion metrics so Hamming should also beat it. Limiting either of them to a random "1.5" makes no sense whatsoever though. Lanczos has 3 lobes, Hamming has 4. Cutting the filter off early makes no sense, specially not when you're doing it in the middle of the second lobe.

[Jules-A]

And this is where the distances are defined, all against the top-left reference f:

That part is obvious, I'm referring to the pp

The only obvious difference is the checkerboard pattern, but I'm not sure if you're talking about this.

I'm not able to spot any major differences on the reference image, I wonder if it's because it's in a higher colour space than usual content.

[Artoriuz]

I'm not able to spot any major differences on the reference image, I wonder if it's because it's in a higher colour space than usual content.

It's not. It's normal sRGB/BT.709.

[Jules-A]

I don't know about limiting FSR to 1.5 but with Hamming I decided to try out image comparison metrics and I don't know if it's just your test image not great at showing worst case scenarios or if I did something wrong since the results seem so close?:

I don't know what you're doing either but FSR has always scored rather poorly. Lanczos already scores higher than it on most distortion metrics so Hamming should also beat it. Limiting either of them to a random "1.5" makes no sense whatsoever though. Lanczos has 3 lobes, Hamming has 4. Cutting the filter off early makes no sense, specially not when you're doing it in the middle of the second lobe.

I don't mean FSR but your version of the FSR lanczos approximation.

I am adjusting the radius in the Hamming code to specifically be 1.5 though. With the version of the algorithm that's currently in use I'm not even sure how.

[Jules-A]

It's not. It's normal sRGB/BT.709.

EDIT: whoops, I should have shown the video version:

The video version of the test image is still 10bit vs 8bit/limited.

[Artoriuz]

I mean no offense but I feel like you're just doing random modifications on things you don't understand. The observations you make are often valuable and they can, sometimes, lead to legitimate issues, but you're often incapable of explaining what exactly you're seeing or what you've done to get there.

It's also a bit difficult to trust anything you provide without double or triple checking since you're usually missing important details (like using multiple shaders, or running low-resolution content, sometimes even the fact that you've modified the shaders in some way).

My point is just that I appreciate the help, but I don't have a crystal ball to read your mind and this is getting exhausting.

About pp, it is just the distance to f.

Also, both mp4s are encoded in 10 bit because that allows x265 to get closer to the reference in the chroma-subsampled case. This doesn't change the colourspace in any way, it just means you have more information to work with. The actual reference.png screenshot taken from 444.mp4 is a normal PNG file with RGBA pixels and 8 bits per channel.

[Artoriuz]

@Jules-A

Pushed this in a different branch: https://github.com/Artoriuz/glsl-chroma-from-luma-prediction/blob/no-spatial-filter/CfL_Prediction_no_spatial_filter.glsl

It doesn't have the internal spatial filter, which means it'll use whatever you set as cscale instead. I don't remember why something similar to this was removed in the past, but I tested it for a grand total of 10 seconds and it seems to be working as intended.

Hopefully this solves your problems as you can set any arbitrary spatial filter with it.

[Jules-A]

I mean no offense but I feel like you're just doing random modifications on things you don't understand. The observations you make are often valuable and they can, sometimes, lead to legitimate issues, but you're often incapable of explaining what exactly you're seeing or what you've done to get there.

I usually do when I submit bug reports but the issue got closed so I didn't think you really cared too much so I cared less about going in to full details and just continued posting my observations from the many experiments/test ect. I did. This issue manifests regardless of MPV settings/scalers used, however it does get worse with scaling. I thought I mentioned that already. Since the issues all appear to be related to the same thing, I didn't bother opening a new bug report.

running low-resolution content

99.9% of the time I'm using the official source from CR/Funimation/Hidive ect. in the max resolution available without any re-encoding. I don't have money to afford Blu-ray copies and honestly expecting people to is silly considering most people probably watch content far worse than the official web source.... Sometimes issues manifest in lower resolution content but completing ignoring that issue because it's low res isn't exactly exceptable.

It's also a bit difficult to trust anything you provide without double or triple checking since you're usually missing important details (like using multiple shaders, or running low-resolution content, sometimes even the fact that you've modified the shaders in some way).

Yeah, it's true, I've modified my shaders a tonne and often forget to mention my settings I tested at, especially lately, however I usually check if I can replicate without other shaders.

[Jules-A]

I don't remember why something similar to this was removed in the past, but I tested it for a grand total of 10 seconds and it seems to be working as intended.

Last time I tested it had offset issues that I couldn't seem to solve, no idea about now.

[Jules-A]

If you want to test the issue I experienced from here: https://github.com/Artoriuz/glsl-chroma-from-luma-prediction/pull/12#issuecomment-1874575639

I cut a few secs from the animation losslessly:

https://github.com/Artoriuz/glsl-chroma-from-luma-prediction/assets/1760158/5072d573-0111-41b1-84fb-5a217342b606

[deus0ww]

Pushed this in a different branch: https://github.com/Artoriuz/glsl-chroma-from-luma-prediction/blob/no-spatial-filter/CfL_Prediction_no_spatial_filter.glsl

It doesn't have the internal spatial filter, which means it'll use whatever you set as cscale instead. I don't remember why something similar to this was removed in the past, but I tested it for a grand total of 10 seconds and it seems to be working as intended.

It's because CHROMA_SCALED is already offset corrected while CHROMA isn't.