Documentation on scaling factor

[Trinity3e]

Hi, I tried setting SF to various values but apparently nothing happens, scale still kicks in, do I need to intervene somehow else?

//

If it works with chroma, can it be made aware of color space so it automatically doubles the scaling vs luma for most 4:2:2 formats? Cause KrigBilateral is broken in gpu-next and nlmeans can replace it this way (even without the bilateral luma-influenced scaling)

//

Also, offtopic, can RF=guided disable itself if the frame rate is high (cause it does take a performance hit then)?

LE2: Patch donut seems to work a bit in temporal, don't remove it yet :-)

[AN3223]

I'm afraid it doesn't after a couple minutes, it variates by +/-3000. On the big PC it's much more stable, within a few hundred

Huh. I'll add writing a benchmark tool to my todo list, it would be much better to measure how long it takes to process a whole video instead of just a short sequence of frames.

I see you pushed a fix, lemme try compilation of everything again

LE: Okay it seems it's working now! LE2: Oh yeah it's all good, i finally synced to my repo the change with temporal luma only + lgc and the guided optimizations. Thanks man.

Awesome!

[AN3223]

Just pushed a commit that reworks the way adaptive sharpening works. I think you will like it. You can compare against the old method by setting ASW=1:ASK=0.

I also changed a bunch of defaults a couple of days ago based on test results, and I will probably make more similar changes soon (please let me know if I screw(ed) anything up).

[Trinity3e]

Certainly, will try later today :)

[Trinity3e]

It's interesting because ASK=1 is practically adaptive sharpening, and can maybe help get rid of adaptive-sharpen.glsl which is sometimes expensive in processing (well as long as the scaling with ewa_lanczos*/spline64 can keep the edges without distorting or haloing them, gotta see that).

What is weird is that, if i put ASK=1, i have to put ASW=0, otherwise it appears too cartoonish, even as i double ASP/F.

ASW=1:ASK=0 behaves exactly like previously, yeah.

Will look around more and report back.

[AN3223]

It's interesting because ASK=1 is practically adaptive sharpening, and can maybe help get rid of adaptive-sharpen.glsl which is sometimes expensive in processing (well as long as the scaling with ewa_lanczos*/spline64 can keep the edges without distorting or haloing them, gotta see that).

It is actually adaptive sharpening, it's just a different way of doing it. Most adaptive sharpeners use a measurement of local contrast to control the sharpening level. The NLM shaders instead use the NLM weight map, a measurement of self-similarity, which is resistant to noise. In addition, NLM uses the NLM blur instead of a box/gaussian blur, which IMO produces a nicer sharpening effect.

The downside to NLM is that it is more expensive to compute. I think if adaptive-sharpen.glsl is slower then that might just be an optimization problem (looks like it doesn't use textureGather, hooks OUTPUT, and does a lot of work with arrays).

What is weird is that, if i put ASK=1, i have to put ASW=0, otherwise it appears too cartoonish, even as i double ASP/F.

Huh, I'm not sure what you are referring to by cartoonish. In my testing ASW=0:ASP=32 produces a much cleaner edge map than ASW=1:ASP=64, but the difference in the actual sharpening effect seems to be negligible.

ASW=1 has higher weights, so maybe ASW=0 with a higher S might produce a better image for you? (I increased the default S by about ~10x recently, you might want to try that)

I notice the effect is way too strong on temporal at very low resolutions (like 360p), I would recommend upscaling (w/ FSRCNNX) prior to NLM to get around this.

If you haven't already, check out the edge map from M=4. I tuned the defaults by looking at that (it is also just fun to look at).

[AN3223]

Speaking of things that are cool to look at, I just added a couple of difference visualization modes (new DV option). They are mostly for debugging but I'm also just having fun with them lol. Try DV=1 on temporal with sharpening (set M=4 on chroma to try it without color).

[Trinity3e]

Hehe, DV=1 looks like a darkroom and DV=2 like the negative when you put alpha in an image processor. :) With M=4 added i can also better see the previous frames of suggestion taken in temporal, really nice! With this occasion i can look if increasing S and ASP tenfold will be okay in every situation.

ASW=1 has higher weights, so maybe ASW=0 with a higher S might produce a better image for you? (I increased the default S by about ~10x recently, you might want to try that)

ASW=1 is okay with ASK=0, the image is probably slightly more natural even, no matter which S/ASP ASK=1 and ASW=0 is also okay-ish but less natural than original

I am still testing with both set to 1 (i guess it will be the best in theory, right?) it does seem indeed i have to increase S/ASP ~tenfold, but i can't say it looks bad at all, it's probably better than before for low quality stuff, it's just more processed. i need to see how to "squeeze" more detail preservation out of it. "post-WD weights" so maybe tweaking WDT/P will do?

I notice the effect is way too strong on temporal at very low resolutions (like 360p), I would recommend upscaling (w/ FSRCNNX) prior to NLM to get around this.

Yeah i was testing with standard def video as i noticed that Do you still recommend using nlmeans pre-upscale in general? as nlmeans_2x is practically post-upscale i guess.

LE: hmm, ASW=1:ASK=1:S=10:ASP=15 looks good if i set WDP higher to 8+ (WD=1 to keep usage low)

[AN3223]

Hehe, DV=1 looks like a darkroom and DV=2 like the negative when you put alpha in an image processor. :) With M=4 added i can also better see the previous frames of suggestion taken in temporal, really nice! With this occasion i can look if increasing S and ASP tenfold will be okay in every situation.

Heh, glad you're enjoying them.

ASW=1 has higher weights, so maybe ASW=0 with a higher S might produce a better image for you? (I increased the default S by about ~10x recently, you might want to try that)

ASW=1 is okay with ASK=0, the image is probably slightly more natural even, no matter which S/ASP ASK=1 and ASW=0 is also okay-ish but less natural than original

I am still testing with both set to 1 (i guess it will be the best in theory, right?) it does seem indeed i have to increase S/ASP ~tenfold, but i can't say it looks bad at all, it's probably better than before for low quality stuff, it's just more processed. i need to see how to "squeeze" more detail preservation out of it. "post-WD weights" so maybe tweaking WDT/P will do?

I think ASW=1 is better in theory, but the edge map looks really bad. For WD=1 it's probably worse. Try comparing ASW=1 to ASW=0 in M=4.

You might want to check against the defaults too, for me ASK=1 is very super obviously better than ASK=0. I only kept ASK=0 around for comparison.

I notice the effect is way too strong on temporal at very low resolutions (like 360p), I would recommend upscaling (w/ FSRCNNX) prior to NLM to get around this.

Yeah i was testing with standard def video as i noticed that Do you still recommend using nlmeans pre-upscale in general? as nlmeans_2x is practically post-upscale i guess.

For low res video I think the sharpening effect is generally better post-upscale (compare edge maps, it's so crisp post-upscale), otherwise I think running NLM pre-upscale is best. Regardless of resolution, softer, low contrast content seems to benefit from pre-upscale sharpening.

I reduced the default ASP to 4 on temporal, I think the sharpening effect is much more pleasant there now. My method for picking the optimal ASP is to view the edge map in M=4 and keep turning the ASP down until the whites are bright.

Also I just pushed a commit adding an option called ASC for softening the edge map, 0 is normal and 1 is linear. I think a low value like ~0.2 might reduce the cartoon effect you were observing, if you're still observing it.

[Trinity3e]

I think ASW=1 is better in theory, but the edge map looks really bad. For WD=1 it's probably worse. Try comparing ASW=1 to ASW=0 in M=4. You might want to check against the defaults too, for me ASK=1 is very super obviously better than ASK=0. I only kept ASK=0 around for comparison.

Okay so ASW=0:ASK=1 it remains unless needing something very HQ :) You may laugh but yesterday as i was looking i thought that your defaults were the other way around, hehe. My bad entirely.

For low res video I think the sharpening effect is generally better post-upscale (compare edge maps, it's so crisp post-upscale), otherwise I think running NLM pre-upscale is best. Regardless of resolution, softer, low contrast content seems to benefit from pre-upscale sharpening.

Yeah i noticed that in analog vhs/ld rips, with careful tuning it works good on digital low res sources as well

I reduced the default ASP to 4 on temporal, I think the sharpening effect is much more pleasant there now. My method for picking the optimal ASP is to view the edge map in M=4 and keep turning the ASP down until the whites are bright.

Optimal calibration :) I'll certainly take the time to do that myself and if i discover other decent combinations i'll let you know :)

Also I just pushed a commit adding an option called ASC for softening the edge map, 0 is normal and 1 is linear. I think a low value like ~0.2 might reduce the cartoon effect you were observing, if you're still observing it.

I think setting it very low like 0.1 makes the image more easy on the eyes regardless, i'm now normally with ASW=0.

[AN3223]

Okay so ASW=0:ASK=1 it remains unless needing something very HQ :) You may laugh but yesterday as i was looking i thought that your defaults were the other way around, hehe. My bad entirely.

Ah okay, that makes sense lol. I probably should have numbered ASW the other way around, it's counterintuitive to have the newer option be the lower number (but that's the way it is so I'll leave it be, changing it now would only make it more confusing).

I reduced the default ASP to 4 on temporal, I think the sharpening effect is much more pleasant there now. My method for picking the optimal ASP is to view the edge map in M=4 and keep turning the ASP down until the whites are bright.

Optimal calibration :) I'll certainly take the time to do that myself and if i discover other decent combinations i'll let you know :)

For sure. DV=1 can be useful for visualizing changes in settings too (any settings, not just AS), so that means you spend less time playing the "corporate needs you to tell the difference between these two images" game!

[AN3223]

I just updated a bunch of defaults. Chroma denoising should be much better now; it turns out it benefits from a lower SW even on light noise. WDT is much lower now too, which should result in slightly better quality and allow S to be more sensitive to changes again. As a result S and ASP are much lower now too. The LGC shaders have had their parameters significantly changed.

Let me know what you think when you get a chance to test. Now I'm focusing on improving HQ and soon I'll work on better upscaling.

[Trinity3e]

I just seen it and wanted to write :) the new defaults seem interesting at being 5 or lower with ASP a bit lower in some which i think are better too. Also curious if removing the rgb hook should maybe lessen usage or improve compatibility, and about shader-test too really professional looking already!

Very quickly dropped in the new LGC, i was watching this analogue vhs rip, it seems to better handle color abberations than before, certainly looks good.

I'll look tomorrow on the rest with great pleasure.

[AN3223]

Also curious if removing the rgb hook should maybe lessen usage or improve compatibility

It will reduce usage by 100% if the video is RGB, since it won't do anything haha. I think that might be for the best though, since most content that is saved as RGB is going to be very high quality anyway.

and about shader-test too really professional looking already!

Heh, thanks.

Very quickly dropped in the new LGC, i was watching this analogue vhs rip, it seems to better handle color abberations than before, certainly looks good.

It doesn't perform very well in the benchmarks (probably because LGC has to deal with noisy luma in the benchmarks, but in real video you might have clean luma and noisy chroma), but I tested it on anime myself and it looks super nice.

[Trinity3e]

It will reduce usage by 100% if the video is RGB, since it won't do anything haha. I think that might be for the best though, since most content that is saved as RGB is going to be very high quality anyway.

Hehe, good that it doesn't trigger anymore with visualizers btw, i'm kinda using mpv as a music player now, it's the only one not buggy on 96khz pw-jack, among all the specialized software out there lol.

Now going back to NLM, so lgc is good, but with the others, I was doing some a-b comparison on the edge maps and it seems a bit too sharp still, both with my settings close-ish to yours and yours entirely. I'm not sure if setting a ~0.2 ASC permanently is a good thing, i don't want to lose detail. Does the ASK map get influenced in any way by the research shapes? LE: on the normal shaders, SW 0.5 on chroma, won't that not longer need a higher S for chroma? Or does it need to be paradoxically higher as a lower weight needs to be denoised more?

[AN3223]

It will reduce usage by 100% if the video is RGB, since it won't do anything haha. I think that might be for the best though, since most content that is saved as RGB is going to be very high quality anyway.

Hehe, good that it doesn't trigger anymore with visualizers btw, i'm kinda using mpv as a music player now, it's the only one not buggy on 96khz pw-jack, among all the specialized software out there lol.

I hadn't thought of that before, yeah that's a good side effect. mpv is a great music player by the way, I think I've been using it since 2019 or 2020 (but I'm not picky).

Now going back to NLM, so lgc is good, but with the others, I was doing some a-b comparison on the edge maps and it seems a bit too sharp still, both with my settings close-ish to yours and yours entirely. I'm not sure if setting a ~0.2 ASC permanently is a good thing, i don't want to lose detail. Does the ASK map get influenced in any way by the research shapes?

ASC shouldn't really reduce detail, it just softens the gradient between sharp/soft regions, I think the effect is very difficult to perceive in practice. Research shapes aren't very relevant to the edge map, there's no direct influence. If the actual image looks too sharp then try turning ASF down, if the edge map looks wrong then try playing with ASP. An ideal edge map should be bright white with black lines on edges, the black areas are the regions that get sharpened.

LE: on the normal shaders, SW 0.5 on chroma, won't that not longer need a higher S for chroma? Or does it need to be paradoxically higher as a lower weight needs to be denoised more?

A lower SW produces more blur, but not that much. You can see there is little difference in the optimal S here:

S=5.208333:WDT=0.750000:SW=1.000000:RF=../guided.glsl 0.93712564545
S=5:WDT=0.750000:SW=0.500000:RF=../guided.glsl 0.938249636

The main difference is that it's better at handling independent specks of noise, they don't have many similar neighbors to blur against so turning S up doesn't really help on its own. The increase in S and the decrease in WDT were because chroma was poorly tuned for a long time.

[AN3223]

Also heads up: I think I'm going to finally rework the cfg system today. The reason is because I want to have a more uniform interface between the cfg scripts for better interoperability. As a side effect it will probably break your Makefile, just letting you know so you're not surprised when you git pull.

[Trinity3e]

Yeah that's good, i needed to rework the parameters anyway since i had them tangled up :)

I hadn't thought of that before, yeah that's a good side effect. mpv is a great music player by the way, I think I've been using it since 2019 or 2020 (but I'm not picky).

One day i'll use some of those fancy osc replacements maybe slimmed down a bit, till then i just put osc always visible and mess around with right click to view the playlist, good enough xd. I pipe the output in a dsp program called stereo tool so that does eq's and stuff, you can see that in my scripts i have one called audio-player-mode. Btw that that script in your folder lilskippa work? I can't get it to work on mine.

ASC shouldn't really reduce detail, it just softens the gradient between sharp/soft regions, I think the effect is very difficult to perceive in practice. Research shapes aren't very relevant to the edge map, there's no direct influence. If the actual image looks too sharp then try turning ASF down, if the edge map looks wrong then try playing with ASP. An ideal edge map should be bright white with black lines on edges, the black areas are the regions that get sharpened.

Okay, got it. Do you recommend DV=2 or DV=1:M=4 to better spot that characteristic halo of oversharpen when it happens? It's very nice that shader-test, i saw you move it from sh to perl (well, raku as called now), can't wrap my head around that now hehe. good job.

The main difference is that it's better at handling independent specks of noise, they don't have many similar neighbors to blur against so turning S up doesn't really help on its own. The increase in S and the decrease in WDT were because chroma was poorly tuned for a long time.

Got it, so from now on i will have chroma S same value as luma. I suppose the 11.66 of lgc is also ideally tuned. :)

[AN3223]

I hadn't thought of that before, yeah that's a good side effect. mpv is a great music player by the way, I think I've been using it since 2019 or 2020 (but I'm not picky).

One day i'll use some of those fancy osc replacements maybe slimmed down a bit, till then i just put osc always visible and mess around with right click to view the playlist, good enough xd. I pipe the output in a dsp program called stereo tool so that does eq's and stuff, you can see that in my scripts i have one called audio-player-mode.

Heh I'm the opposite, I don't use any osc and I just use m3u files or --shuffle to handle playlists (I don't listen to much music though, so I'm fine with it being sub-optimal). Sometimes I use avectorscope for audio visualization, I have that bound in my input.conf, but the CPU usage is kinda high.

Btw that that script in your folder lilskippa work? I can't get it to work on mine.

It works but it doesn't work very well, you're not really missing out. I never use it. Does it just keep doing fast forward until you cancel it? The scene detection will do that a lot.

Okay, got it. Do you recommend DV=2 or DV=1:M=4 to better spot that characteristic halo of oversharpen when it happens?

Like this? I think that effect doesn't really appear in NLM because of the edge preserving blur, but it does sometimes amplify noise in edge areas when it is present. I'm not sure how useful DV may or may not be for sharpening, I would just say try it out and see if you see anything useful?

It's very nice that shader-test, i saw you move it from sh to perl (well, raku as called now), can't wrap my head around that now hehe. good job.

It's a fun language to work with, this is my first time doing anything with it, and I never wrote much Perl before. I otherwise would have used Python, but Raku is so expressive and so far I feel like it was a good choice. Stuff like this and this are really fun to write and not difficult to understand once you know how the operators work, I don't find them any more difficult to understand than the nested for-loops that they replace.

Got it, so from now on i will have chroma S same value as luma. I suppose the 11.66 of lgc is also ideally tuned. :)

Huh, I feel chroma should be higher since it is a simpler case and the human eye isn't very sensitive to loss of chroma detail, but it is subjective after all, so I'm not telling you that you are wrong or anything.

The denoising factors are all based on the optimal values found in testing, but the dataset isn't perfect and the noise model isn't perfect either, and I think that's especially true for LGC. In testing, LGC has to denoise chroma from a noisy chroma plane while using a noisy luma plane as guidance. In reality the luma plane might not be as noisy as the chroma plane, so in that case LGC could benefit from an even stronger blur? But I feel like the effect is strong enough as is, I would be hesitant to turn the default up any further.

[Trinity3e]

I rebased today, works fine, I notice way less now the oversharpened edges, not sure if thanks to the recent change of yesterday or me fixing parameters, but it's good.

Btw that that script in your folder lilskippa work? I can't get it to work on mine. It works but it doesn't work very well, you're not really missing out. I never use it. Does it just keep doing fast forward until you cancel it? The scene detection will do that a lot.

Hmm, first time i tested it didn't do anything, now i tried in this 1 hour film, says skipping to black/scene but seems to freeze.

Like this? I think that effect doesn't really appear in NLM because of the edge preserving blur, but it does sometimes amplify noise in edge areas when it is present. I'm not sure how useful DV may or may not be for sharpening, I would just say try it out and see if you see anything useful?

Yeah, like that, after having a look more at it seems to help a bit in spotting but for sharpening ultimately you have to eye it out indeed. :)

It's a fun language to work with, this is my first time doing anything with it, and I never wrote much Perl before. I otherwise would have used Python, but Raku is so expressive and so far I feel like it was a good choice. Stuff like this and this are really fun to write and not difficult to understand once you know how the operators work, I don't find them any more difficult to understand than the nested for-loops that they replace.

Hehe, looks fun. Python to me looks like a interbreed of script and program, while Perl/Raku looks pretty, but of course as i am only used to scripting, a bit tangly :) glad that you found it to suit your need this way. one day when bored make a documentation for shader-test too :)

Huh, I feel chroma should be higher since it is a simpler case and the human eye isn't very sensitive to loss of chroma detail, but it is subjective after all, so I'm not telling you that you are wrong or anything. The denoising factors are all based on the optimal values found in testing, but the dataset isn't perfect and the noise model isn't perfect either, and I think that's especially true for LGC. In testing, LGC has to denoise chroma from a noisy chroma plane while using a noisy luma plane as guidance. In reality the luma plane might not be as noisy as the chroma plane, so in that case LGC could benefit from an even stronger blur? But I feel like the effect is strong enough as is, I would be hesitant to turn the default up any further.

Got it. And regarding the params don't worry, i only look subjectively for detail, just wanted the settings to look clean :)

I think that the lower SW for chroma on the normal shaders indeed benefits as you also noticed, looks smoother on lq for hd stuff

Btw speaking of that, i still find temporal a bit too expensive for reliably playing hd video, i only have it set for 720p <=30fps, for 1080 usage jumps a bit.

[AN3223]

I rebased today, works fine, I notice way less now the oversharpened edges, not sure if thanks to the recent change of yesterday or me fixing parameters, but it's good.

I didn't change much so it was probably just you. The breaking change I mentioned before isn't part of the build system yet, I decided I would bite the bullet and rewrite the whole cfg system since it would be better in the long run, and that's still a WIP.

Btw that that script in your folder lilskippa work? I can't get it to work on mine. It works but it doesn't work very well, you're not really missing out. I never use it. Does it just keep doing fast forward until you cancel it? The scene detection will do that a lot.

Hmm, first time i tested it didn't do anything, now i tried in this 1 hour film, says skipping to black/scene but seems to freeze.

Does mpv happen to freeze if you just run it with --video-sync=desync --speed=100?

Btw speaking of that, i still find temporal a bit too expensive for reliably playing hd video, i only have it set for 720p <=30fps, for 1080 usage jumps a bit.

Yeah temporal needs more work. I have some ideas but I want to focus on upscaling first, and before that I need a more robust build system, and I have some other stuff I might want to do in the interim.

[AN3223]

one day when bored make a documentation for shader-test too :)

Sorry I just realized that I forgot to respond to this. This is planned. Right now all of the options are documented but I understand the general usage and purpose of shader_test is somewhat ambiguous, I will clarify this more in the future. Currently it's in a somewhat early stage and very subject to change, so documentation would be quickly become obsolete (the build system rework is related to this too, testing NLM upscaling is broken in shader_test because of shortcomings of nlmeans_cfg).

[Trinity3e]

Hi, i've seen the new buildsystem and makefile convention, really clean looking, and many recent defaults are kinda close to what i use so i barely override a few params now, all jolly well, but i can't seem to make it work with ./Makefile.nlm -B awk: ./shader_cfg:78: error: function `hooks': cannot use function name as parameter name // with unedited makefile I suppose gnu vs nongnu awk again :) will try with mobile telephone running alpine later.

Sorry I just realized that I forgot to respond to this. This is planned. Right now all of the options are documented but I understand the general usage and purpose of shader_test is somewhat ambiguous, I will clarify this more in the future. Currently it's in a somewhat early stage and very subject to change, so documentation would be quickly become obsolete (the build system rework is related to this too, testing NLM upscaling is broken in shader_test because of shortcomings of nlmeans_cfg).

Oh yeah so after buildsystem works good you plan to work on upscaling, temporal and the rest of shader-test, good luck, looking forward to it :) Possibly you might think about a hybrid EP with FSRCNNX so that it doesn't run pre denoise, can increase efficiency just enough maybe?

[AN3223]

Ah shoot, I didn't test the recent changes against gawk. I'll fix it now

[AN3223]

Wow the default return value of a function in gawk is 0 but in busybox awk it's 1

[Trinity3e]

lol, like an error busybox sed -E also kinda behaves like it had a bit too much brandy :) don't know why honestly, i admire it's tiny size but some things get mixed up when trying to do compatibility with gnu systems

[AN3223]

Okay it should be fixed now. I'm kinda surprised some of these things weren't issues in the first place lol.

[AN3223]

Oh yeah so after buildsystem works good you plan to work on upscaling, temporal and the rest of shader-test, good luck, looking forward to it :)

Actually I had a couple of easy to implement ideas while I was working on shader_cfg, and I've also been thinking about how poorly tuned LQ and HQ are. And I want to bring back anime denoising. So some smaller cool things will be coming before the bigger cool things.

Possibly you might think about a hybrid EP with FSRCNNX so that it doesn't run pre denoise, can increase efficiency just enough maybe?

I'm not sure what you mean. EP is just a downscale stage, it has very little impact on performance.

BTW shader_cfg can't properly build Guided LGC right now (I imagine you're probably using NLM LGC instead anyways)

[Trinity3e]

Okay it should be fixed now. I'm kinda surprised some of these things weren't issues in the first place lol.

:) It works now, but weirdly some videos make mpv crash now, it didn't before. It spits out nothing except sigsegv. Can you have a look quickly over params i've put? https://gist.github.com/Trinity3e/8f6bea626798e5e30981bbf92ed2082f

Actually I had a couple of easy to implement ideas while I was working on shader_cfg, and I've also been thinking about how poorly tuned LQ and HQ are. And I want to bring back anime denoising. So some smaller cool things will be coming before the bigger cool things.

Neat. About anime, I've been watching some old bad quality one recently and temporal does such a good job when AS is on that i honestly wouldn't see a need, like the edges are perfect, looking from up close on the 4k tv, and that comes from a really badly mastered 480p bootleg dvd.

I'm not sure what you mean. EP is just a downscale stage, it has very little impact on performance.

Sorry, i meant when FSRCNNX is injected

BTW shader_cfg can't properly build Guided LGC right now (I imagine you're probably using NLM LGC instead anyways)

I guess, whatever the default is. I'd welcome a lower usage one to maybe use at 4k non hdr (or wait, hdr might work too, cause the tone mapping is before glsl afaik :) )

[AN3223]

Okay it should be fixed now. I'm kinda surprised some of these things weren't issues in the first place lol.

:) It works now, but weirdly some videos make mpv crash now, it didn't before. It spits out nothing except sigsegv. Can you have a look quickly over params i've put? https://gist.github.com/Trinity3e/8f6bea626798e5e30981bbf92ed2082f

I've been running into segfaults too whenever I'm chaining too many shaders together. Especially FSRCNNX-16 + anything else. I'm not sure why, I ought to debug it. It might have something to do with the large amount of shader stages.

I should have mentioned this before, I changed the way the options are passed to the script. So OPTS_CHROMA should actually be OPTS_OTHER. Or better yet, just pass the chroma opts to OPT and then pass the luma opts to OPTS_LUMA (you can see that's the way it is done in the defaults).

Also don't set EP=1, it's a scaling factor, you probably want EP=3/ (which will set the EP macro to 1).

Actually I had a couple of easy to implement ideas while I was working on shader_cfg, and I've also been thinking about how poorly tuned LQ and HQ are. And I want to bring back anime denoising. So some smaller cool things will be coming before the bigger cool things.

Neat. About anime, I've been watching some old bad quality one recently and temporal does such a good job when AS is on that i honestly wouldn't see a need, like the edges are perfect, looking from up close on the 4k tv, and that comes from a really badly mastered 480p bootleg dvd.

Huh, I haven't seen any anime where AS looks good, I always notice it amplifying noise near edges. Yesterday I was watching some anime with the default profile and I was very impressed with how well it handles compression noise in most areas.

I'm not sure what you mean. EP is just a downscale stage, it has very little impact on performance.

Sorry, i meant when FSRCNNX is injected

I'm still confused

BTW shader_cfg can't properly build Guided LGC right now (I imagine you're probably using NLM LGC instead anyways)

I guess, whatever the default is. I'd welcome a lower usage one to maybe use at 4k non hdr (or wait, hdr might work too, cause the tone mapping is before glsl afaik :) )

I mean guided_lgc won't work if you try to build it yourself, but the one that is included in the git repo works fine. It does offer lower GPU usage too, and a similar effect to NLM LGC.

[Trinity3e]

I should have mentioned this before, I changed the way the options are passed to the script. So OPTS_CHROMA should actually be OPTS_OTHER. Or better yet, just pass the chroma opts to OPT and then pass the luma opts to OPTS_LUMA (you can see that's the way it is done in the defaults).

Got it, did so.

Also don't set EP=1, it's a scaling factor, you probably want EP=3/ (which will set the EP macro to 1).

Exactly like that? I was just trying to match the luma EP settings to chroma.

Huh, I haven't seen any anime where AS looks good, I always notice it amplifying noise near edges. Yesterday I was watching some anime with the default profile and I was very impressed with how well it handles compression noise in most areas.

True, i do forget that my monitor is a 4k tv so i probably don't notice fine noise, but I think too much spatial denoising can affect if a cartoon has some real images in it, will pastel them, correct me if i'm wrong.

I'm still confused

:) I mean if it changed what you refered to previously last month:

I still don't think you will be impressed. The quality is worse than nlmeans+FSRCNNX and the performance is worse too since NLM is running at 2x the resolution.

/////

gol.glsl is cool, based on it you can make a nice effect framework in glsl one day :)

/////

Unfortunately the current temporal still segfaults my mpv. The temporal i had saved from 3 weeks ago in my repo works fine. If you want and have the time, please diff them, maybe you see something https://gitlab.com/Hitman_47/mpv_config/-/blob/main/mpv/shaders/nlmeans_temporal.glsl

[AN3223]

Also don't set EP=1, it's a scaling factor, you probably want EP=3/ (which will set the EP macro to 1).

Exactly like that? I was just trying to match the luma EP settings to chroma.

Yes, EP=3/ means "downscale by a factor of 3" which is the default for luma EP.

Huh, I haven't seen any anime where AS looks good, I always notice it amplifying noise near edges. Yesterday I was watching some anime with the default profile and I was very impressed with how well it handles compression noise in most areas.

True, i do forget that my monitor is a 4k tv so i probably don't notice fine noise, but I think too much spatial denoising can affect if a cartoon has some real images in it, will pastel them, correct me if i'm wrong.

I actually recently found that the default SS=0.25 works just as well on anime as it does for any other content, or at least it does in shader_test. I used to believe SS=0.0 was better for anime but now I think that's just not correct.

I feel like noise should be more noticeable on a 4k TV, unless the TV is doing some kind of post-processing itself (mine has some kind of built-in sharpening/denoising upscaler by default, so I imagine it's not uncommon).

I'm still confused

:) I mean if it changed what you refered to previously last month:

I still don't think you will be impressed. The quality is worse than nlmeans+FSRCNNX and the performance is worse too since NLM is running at 2x the resolution.

I did improve scaling recently, nlmeans_2x certainly looks better than it did before, but it's not worth using quite yet. It's going to require a lot of testing (and probably modifications to shader_test). The current implementation of using FSRCNNX as a guide image is just a crutch, NLM should be able to produce a nice effect on its own.

/////

gol.glsl is cool, based on it you can make a nice effect framework in glsl one day :)

/////

Yeah that was lots of fun to make. I think it would be easy to extend it to do make a simple smoke-like effect. I was also thinking I would make a VHS effect based on Guided Filter.

Unfortunately the current temporal still segfaults my mpv. The temporal i had saved from 3 weeks ago in my repo works fine. If you want and have the time, please diff them, maybe you see something https://gitlab.com/Hitman_47/mpv_config/-/blob/main/mpv/shaders/nlmeans_temporal.glsl

Sorry, a lot has changed so it would be difficult to narrow it down. Could you try building the current temporal with RF=0:RF_LUMA=0:EP=0:WD=0:NG=1:SST=0? If that works, try to see which feature is breaking things so maybe we can do something about it. If that doesn't work, try hand editing the shader to replace r16f with r32f and see if that helps.

Are the temporal shaders the only ones causing segfaults for you?

[Trinity3e]

NG=1 got further now, mpv shows 1 frame then segfaults, the other parameters have same result as before

hqsharpen+denoise also doesn't work, now i saw when testing

LQ and LGC work fine.

[AN3223]

Could you possibly track down a specific commit that introduces the segfaults?

[Trinity3e]

Yes i will try, should be one from roughly 2 weeks ago

I feel like noise should be more noticeable on a 4k TV, unless the TV is doing some kind of post-processing itself (mine has some kind of built-in sharpening/denoising upscaler by default, so I imagine it's not uncommon).

I'm sitting kinda far away from mine, 55 inches 3 meters away. I turned off all it's processing and sensors from the samsung factory menu, it's basically turned into a monitor

I was also thinking I would make a VHS effect based on Guided Filter.

Oh yeah, please make that, and also a CRT shadow mask abberation effect based on the same thing! :-)

[Trinity3e]

Okay, this is getting weirder, with the default settings versions from a month ago also segfault. I will need to track down a backup of my home directory where i had those exact settings from when mine worked, and build versions from ~2 weeks ago with those parameters This will take a while i guess

Is there a way to only git clone your mpv directory, cause i will be doing a lot of those :)

[AN3223]

Okay, this is getting weirder, with the default settings versions from a month ago also segfault. I will need to track down a backup of my home directory where i had those exact settings from when mine worked, and build versions from ~2 weeks ago with those parameters This will take a while i guess

Huh okay. I'm able to get a segfault just from running NLM twice, but running it once works just fine. Rebuilding with RF=0:RF_LUMA=0 the segfault goes away. So it does seem like it might have something to do with the large amount of shader stages.

I built libplacebo & mpv masters with debug symbols and I'm running them in gdb now. I can see the segfault is coming from libplacebo and it only occurs with --vo=gpu-next (can you confirm this?). I'll gather more info and open an issue there.

Is there a way to only git clone your mpv directory, cause i will be doing a lot of those :)

AFAIK, no. But you shouldn't have to clone more than once, just do a regular (non-shallow) clone and then git checkout to whatever commit you want. But you don't need to track down the commit, it's probably not worth the trouble.

[AN3223]

I narrowed it down to guided.glsl and opened an issue https://code.videolan.org/videolan/libplacebo/-/issues/272

[Trinity3e]

That's a good find, would've taken me ages to narrow it down! Really appreciate it. Indeed it only happens in gpu-next.

A guy there said there's this patch not approved yet, will try to build mpv master with it to see

BTW the issue is kinda getting close to 100 replies and it won't look pretty, i'll email you from now on if that's fine.

[AN3223]

A guy there said there's this patch not approved yet, will try to build mpv master with it to see

Thank you for letting me know, I didn't get a timely notification for that comment. I just tried the PR and it does work!

BTW the issue is kinda getting close to 100 replies and it won't look pretty, i'll email you from now on if that's fine.

I would prefer to keep it here, since it's about the project.

[Trinity3e]

Thank you for letting me know, I didn't get a timely notification for that comment. I just tried the PR and it does work!

Yep, I got to build mpv master with that patch today and it does work! Strangley enough I don't get the weird text you got when loading guided twice. And my gpu is Navi with it's still borked drivers

/// Before that latest patch with LGC, was that also scaling chroma to full size right away? Or it needed/still needs shader opts?

[AN3223]

Thank you for letting me know, I didn't get a timely notification for that comment. I just tried the PR and it does work!

Yep, I got to build mpv master with that patch today and it does work! Strangley enough I don't get the weird text you got when loading guided twice. And my gpu is Navi with it's still borked drivers

That part is fixed now, the PR was probably updated by the time you got to it :)

/// Before that latest patch with LGC, was that also scaling chroma to full size right away? Or it needed/still needs shader opts?

It was upscaling chroma before, but it looked bad after the spatial rework since there wasn't enough blur. It should just be a parameter issue w/o needing changes to the shader code, but testing it properly will require modifications to shader_test so I disabled LGC upscaling in the meantime.

[AN3223]

Hey I just thought I would let you know I just pushed a commit that greatly improves the sharpening effect, let me know what you think!

[Trinity3e]

Looks good! I think the edges look the best so far on the analogue sources. I'll have a lash later how temporal looks, you know, needs that patch :-) Nice idea you had also on nlmeans_2x to use lanczos baselayer for the scaling instead of FSR, reduces usage to an acceptable -hq equivalent and looks quite nice still.

[AN3223]

Looks good! I think the edges look the best so far on the analogue sources. I'll have a lash later how temporal looks, you know, needs that patch :-)

Give it a try on anime. For me the difference is night and day, I really didn't like the sharpening filters on anime before. Has some issues with ringing though, I'll look into that later.

Nice idea you had also on nlmeans_2x to use lanczos baselayer for the scaling instead of FSR, reduces usage to an acceptable -hq equivalent and looks quite nice still.

It's still worse than just using lanczos though. Increasing S gets it to be practically identical to lanczos but decreasing S decreases the quality. Not sure what I'm doing wrong, it will take more investigation.

[Trinity3e]

Yeah it's strikingly universal with all default values, seems that barely any tweaking will be even needed, of course I need to learn using the new shader-test and some input from you when further tweaking the values :-) Also fortunately mpv doesn't take too long time to rebuild The ringing isn't too noticeable, it's comparable to before if rotations off and ASP a bit high (but of course without the oversharpening halos which don't show up at all now)

[Trinity3e]

Hi, a while has passed, i think you saw that haasn merged the commit at my suggestion, so now i can properly install mpv-build-git (arch equivalent of mpv-master)

I have this issue when building temporal, it throws this error. The other temporal shaders from the repo are fine.

[vo/gpu-next/libplacebo] shader compile log (status=0): 0:984(13): error: `PREV2' undeclared
[vo/gpu-next/libplacebo] 0:984(51): error: `PREV2' undeclared
[vo/gpu-next/libplacebo] 0:984(41): error: no matching function for call to `imageSize(error)'; candidates are:
[vo/gpu-next/libplacebo] 0:984(41): error:    int imageSize(image1D)
...
[vo/gpu-next/libplacebo] 0:984(26): error: operands to arithmetic operators must be numeric
[vo/gpu-next/libplacebo] 0:984(20): error: cannot construct `ivec2' from a non-numeric data type
[vo/gpu-next/libplacebo] 0:984(2): error: no matching function for call to `imageStore(error, error, vec4)'; candidates are:
[vo/gpu-next/libplacebo] 0:984(2): error:    void imageStore(image1D, int, vec4)
...

Only guided loads.

[AN3223]

If I understand correctly, the temporal shaders I built are fine but the ones you built aren't fine? Can you show me your Makefile?

[Trinity3e]

Yes. https://gist.github.com/Trinity3e/c0bfb7b4ca5797f5da6a83182fc81080 Just the bottom of the file I replace, got used to this so I can easily adapt to changes, maybe I might've overlooked one of your changes with subdirectories by quality.

What's missing is:

//!BIND PREV2

    case 2: return val_swizz(imageLoad(PREV2, ivec2((HOOKED_pos + HOOKED_pt * vec2(off)) * imageSize(PREV2))));

//!TEXTURE PREV2
//!SIZE 1920 1080
//!FORMAT r16f
//!STORAGE

[AN3223]

You're setting T=3 but still using the default T_FRAME_LIMIT of 2, I'm guessing. It's like an array OOB error.

[Trinity3e]

Hmm Tried, to no success, it will then complain of PREV3 missing T=2 closes with the same error