FineDehalo produces white outlines p2

[nobananasforyou]

Hello,

I've grabbed your latest commit and am running vs r50, and yet I am still seeing white outlines. I've done an image comparison.

src > avs > vs, in that order: https://slow.pics/c/ch3n9DXr

Alternatively, you can download the images from dropbox: https://www.dropbox.com/s/3vm9rkxpdplzky0/s5.rar?dl=0

Here's the vs script: import vapoursynth as vs import havsfunc as haf import fvsfunc as fvf core = vs.get_core()

src = core.d2v.Source(r'sad', rff=False) src = core.std.ShufflePlanes(src, planes=0, colorfamily=vs.GRAY) src = fvf.Depth(src, 16) src = haf.FineDehalo(src, darkstr=0) src = fvf.Depth(src, 10) src.set_output()

avs script: MPEG2Source("sad.d2v", cpu=0) finedehalo(darkstr=0)

[theChaosCoder]

If you show the mask, this is what it looks like https://i.imgur.com/MFUG6Bu.png For 8-12bit the mask looks the same.

Usually cpu none = not broken :D

clip = mvf.Depth(clip, 16)
clipx = haf.FineDehalo(clip, rx=2, darkstr=0, showmask=True).text.Text("16bit")
core.std.SetMaxCPU("none")
clip = haf.FineDehalo(clip, rx=2, darkstr=0, showmask=True).text.Text("16bit CPU none")
clip= core.std.StackHorizontal([clip, clipx])

Seems that sobel() has an issue https://github.com/vapoursynth/vapoursynth/issues/574#issuecomment-632738189

But this does not explain the differences between vs and avs.

[theChaosCoder]

I compared the output line for line and the biggest difference starts here. Look at the edges. https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/d0933b4f9ec7430ac2019f0ad96e5879f8ed387c/havsfunc.py#L552

# This mask is almost binary, which will produce distinct discontinuities once applied. Then we have to smooth it
    shrink = shrink.std.Convolution(matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1]).std.Convolution(matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])
    return shrink

[sekrit-twc]

The input to std.Convolution already has those extra edges. Are you sure the problem begins there?

[theChaosCoder]

You are right. I skipped one line xD It begins with the line above with mt_inpand_multi(shrink, mode='ellipse', sw=rx_i, sh=ry_i)

[sekrit-twc]

Isn't it because mt_inpand_multi maps to std.Minimum, but the original is something else?

    (mode_m != "") ? src.mt_expand (
\       thY=thY, thC=thC, mode=mode_m,
\       offx=offx, offy=offy, w=w, h=h, y=y, u=u, v=v, chroma=chroma
\   ).mt_expand_multi (
\       thY=thY, thC=thC, mode=mode,
\       offx=offx, offy=offy, w=w, h=h, y=y, u=u, v=v, chroma=chroma,
\       sw=sw-1, sh=sh-1
\   ) : src

[theChaosCoder]

I found this https://github.com/pinterf/masktools/issues/11

Btw VS and AVS+ mt_expand/inpand_multi have identical output ans same speed.

I replaced mt_inpand_multi with the AVS version (with help of avsw.Eval) and it looks the same :/

[theChaosCoder]

Idk maybe the issue starts with the first edge mask. The avs version looks like something between sobel and prewitt. I think it looks a bit better if I replace edges = src.std.Sobel() with edges = src.std.Prewitt()

[sekrit-twc]

Could be running into the issue where masktools Prewitt/Sobel/etc. are something else entirely:

http://avisynth.nl/index.php/MaskTools2/Mt_edge

             "prewitt" is a more robust kernel and is equivalent to: 

            mt_logic(mt_logic(mt_edge("1 1 0 1 0 -1 0 -1 -1 1"),mt_edge("1 1 1 0 0 0 -1 -1 -1 1"),mode="max"),mt_logic(mt_edge("1 0 -1 1 0 -1 1 0 -1 1"),mt_edge("0 -1 -1 1 0 -1 1 1 0 1"),mode="max"),mode="max")

[HolyWu]

I will be busy for a while so I don't have much time to hunt down the culprit now. But neither "sobel" nor "prewitt" in AviSynth mt_edge's definition adheres to the standard operators, as already have been mentioned at https://forum.doom9.org/showthread.php?p=1839201#post1839201.

[theChaosCoder]

But as a first fix Sobel could be replaced with Prewitt as it seems to be closer to avisynths "prewitt".

Here's the same shrink output from above. I only replaced Sobel with Prewitt. Look at the hole in the nose, even that is now visible

EDIT (same img from previous comment)

Avisynth code

# Basic edge detection, thresholding will be applied later.
edges = defined(edgemask) ? edgemask : mt_edge (mode="prewitt", thY1=0, thY2=255)

[theChaosCoder]

@HolyWu You actually ported avs prewitt already: http://forum.doom9.net/showthread.php?p=1839201#post1839201 But the difference is not that big between std.Prewitt and "avs Prewitt".

# mt_edge("prewitt", 0, 255)
clip1 = core.std.Expr([core.std.Convolution(clip, matrix=[1, 1, 0, 1, 0, -1, 0, -1, -1], saturate=False), core.std.Convolution(clip, matrix=[1, 1, 1, 0, 0, 0, -1, -1, -1], saturate=False)], 'x y max')
clip2 = core.std.Expr([core.std.Convolution(clip, matrix=[1, 0, -1, 1, 0, -1, 1, 0, -1], saturate=False), core.std.Convolution(clip, matrix=[0, -1, -1, 1, 0, -1, 1, 1, 0], saturate=False)], 'x y max')
clip = core.std.Expr([clip1, clip2], 'x y max')

[HolyWu]

I replaced Sobel with AvsPrewitt. See if it's better now.

[nobananasforyou]

It's a little bit better but it still results in halos around the eyes and nose.

[theChaosCoder]

I noticed that Dehalo_alpha blurs the img more then the avs version All I did was to return the halos var https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/39c2e9dc6604658600210c88717c6c56f5efa3d2/havsfunc.py#L413 I checked both m4() results, they match with avs So why does it blur more?

EDIT Argh I forgot to uncomment my vivtc.VFM(0) line Now avs dehalo is a bit more blurry

[HolyWu]

Ah...the Bicubic default in zimg has changed to Catmull-Rom instead of Mitchell-Netravali, causing the result not as blurry as before. Try the latest commit again.

[theChaosCoder]

I would say it is very close to avs finedehalo now. FineDehalo(clip, rx=2, darkstr=0)

EDIT wrong output. Here is the correct one

[theChaosCoder]

But I wonder which avs ports are also affected by this bicubic change in zimg...

[theChaosCoder]

Would this line also be affected in FineDehalo2() ? https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py#L676

I think there are only 2 potential other candidates bbmod https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py#L3616 and GrainFactory3 https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py#L3749

[HolyWu]

It doesn't matter in FineDehalo2 since Bicubic is only used for converting gray mask to the input format when the user wants to show the mask.

bbmod and GrainFactory3 have explicitly specified filter_param_a and filter_param_b so they are fine as well.

[nobananasforyou]

@HolyWu your latest commit resolves this issue. Thank you so much. Cheers!