Cdeblend weird behaviour

[theChaosCoder]

I don't know how to describe it, but it seems that cdeblend sometimes do "nothing", like it's not really deblending at all.

with omode=4 it always shows 0.0. But for some reason it worked once and showed the correct values + BLEND string

I restarted vseditor multiple times, tried to seek from frame 0. Nothing seems to help. The avisynth version worked correctly in all my tests.

I have no idea what the issue could be. Can you reproduce the issue? Python 3.10, VS R59, Windows 10 x64

clip = clip.vivtc.VFM(0)
vs=muf.Cdeblend(clip).text.Text("Cdeblend Vapoursynth")
avs = core.avsw.Eval('cdeblend()', clips=[clip], clip_names=["last"]).text.Text("Cdeblend Avisynth")
clip=core.std.StackHorizontal([vs, avs])

Test file https://www.dropbox.com/s/kl7llfc2vsy29l4/blend%20_sample-001.mkv?dl=1

[WolframRhodium]

Thanks, I will test it later.

[Selur]

might be related to that std.Cache filter becomes no-op in API4. got a similar issue with sRestore, see: https://forum.doom9.org/showthread.php?p=1986339

[WolframRhodium]

Thanks, I think it should be simply labeled as "unusable", because the requests are never linear.

[theChaosCoder]

UPDATE maybe you saw my posts in discord.

There are 2 Problems:

1. vsedit, because of how it requests frames, omode=4 shows always 0.0. This does not happen in VS-Preview! It is also ok if you make an actuall encode...
2. you can get linear behaviour if you set core.num_threads=1

I compared R52 vseditor with R63 vspreview and the blend removal / quality is better in R52. Not sure why...

[WolframRhodium]

Please try update muvsfunc.py (https://github.com/WolframRhodium/muvsfunc/commit/0fe9af57312534b85db57240472196d3cc071974) and check whether setting sequential=True works.

[theChaosCoder]

vsedit seems to work now, even backwards seeking seems to be reliable. I get stats with omode=4 But now vspreview does not work anymore, it is stuck on loading or something (does not even show the gui). It's always something... xD

Tested with Cdeblend(clip, sequential=True)

python -m vspreview D:\cdeblend_test.vpy

[WolframRhodium]

I notice similar behaviour on vspreview.

It seems to relate to the length of clip, since it appears to work on clips with no more than a few hundreds of frames.

[WolframRhodium]

Speed test indicates that

export N=999 # any number
vspipe --start $N --end $N script.vpy .

is much slower than

vspipe --start 0 --end $N script.vpy .

(680 fps v.s. 0.02 fps with N=400)

This may be the effect of frame caching, so it might be preferred to do sequential previewing even though the result is correct.

For vspreview it seems to require manually setting frame_to_show and last_showed_frame variables to zero in .vspreview/script.yml.

[theChaosCoder]

680 vs 0.02 fps is a HUGE difference lol

[myrsloik]

And there we go. Output (current variable) identical in my test clip. If you somehow find a clip where it's not identical and it actually matters just try setting preroll to 19, 24 or 29. Report your findings.

def CdeblendPrime(input: vs.VideoNode, omode: int = 0, bthresh: float = 0.1, mthresh: float = 0.6,
             xr: float = 1.5, yr: float = 2.0, fnr: bool = False,
             dclip: Optional[vs.VideoNode] = None
             ) -> vs.VideoNode:

    funcName = 'CdeblendPrime'

    # how many frames before the current one metrics are computed for, must be 1 or bigger, consistent results start around 14
    preroll = 14

    # check
    if not isinstance(input, vs.VideoNode):
        raise TypeError(f'{funcName}: "input" must be a clip!')

    if dclip is None:
        if input.format.color_family not in [vs.YUV, vs.GRAY]:
            raise TypeError(f'{funcName}: "input" must be a YUV/Gray clip if "dclip" is not provided!')
    else:
        if not isinstance(dclip, vs.VideoNode) or dclip.format.color_family not in [vs.YUV, vs.GRAY]:
            raise TypeError(f'{funcName}: "dclip" must be a YUV/Gray clip!')

        if dclip.num_frames != input.num_frames:
            raise TypeError(f'{funcName}: "dclip" must of the same number of frames as "input"!')

    preproc = {i: input[0:2] + input[2+i:] for i in range(-2, 3)}

    def evaluate(n: int, f: List[vs.VideoFrame], clip: vs.VideoNode, core: vs.Core) -> vs.VideoNode:
        # coefficients
        Cbp2 = 128.0 if bthresh > 0 else 2.0
        Cbp1 = 128.0 if bthresh > 0 else 2.0
        Cbc0 = 128.0 if bthresh > 0 else 2.0
        Cbn1 = 128.0 if bthresh > 0 else 2.0
        Cbn2 = 128.0 if bthresh > 0 else 2.0

        Cdp1 = mthresh * 0.5
        Cdc0 = mthresh * 0.5
        Cdn1 = mthresh * 0.5

        current = 0

        def computestate(Cdiff: float, Cbval: float):
            nonlocal Cdp1, Cdc0, Cdn1, Cbp2, Cbp1, Cbc0, Cbn1, Cbn2, current

            Cdp1 = Cdc0
            Cdc0 = Cdn1 if omode > 1 else Cdiff # type: ignore
            Cdn1 = Cdiff # type: ignore

            Cbp2 = Cbp1
            Cbp1 = Cbc0
            Cbc0 = Cbn1 if omode > 1 else (0.0 if Cdc0 < mthresh else (Cbval - Cbn2) / ((max(Cdc0, Cdp1) + mthresh) ** 0.8)) # type: ignore
            Cbn1 = 0.0 if (Cdc0 < mthresh or Cdn1 < mthresh) else (Cbval - Cbn2) / ((max(Cdc0, Cdp1) + mthresh) ** 0.8) # type: ignore
            Cbn2 = Cbval # type: ignore

            current = (1 if ((Cbn1 < -bthresh and Cbc0 > bthresh) or (Cbn1 < 0 and Cbc0 > 0 and Cbc0 + Cbp1 > 0 and Cbc0 + Cbp1 + Cbp2 > 0)) else
                       0 if ((Cbc0 < -bthresh and Cbp1 > bthresh) or (Cbc0 < 0 and Cbc0 + Cbn1 < 0 and Cbp1 > 0 and Cbp1 + Cbp2 > 0)) else
                       (2 if Cbn1 > 0 else 1) if current == -2 else
                       current - 1)

            if omode == 2:
                current = (-1 if min(-Cbp1, Cbc0 + Cbn1) > bthresh and abs(Cbn1) > abs(Cbc0) else
                           1 if min(-Cbp2 - Cbp1, Cbc0) > bthresh and abs(Cbp2) > abs(Cbp1) else
                           -1 if min(-Cbp1, Cbc0) > bthresh else
                           0)

            if omode <= 1:
                current = (0 if min(-Cbp1, Cbc0) < bthresh else
                           -1 if omode == 0 else
                           1)

        for i in range(max(0, preroll - n), preroll + 1):
            Cdiff = f[i * 2 + 0].props["PlaneMAE"] * 255 # type: ignore
            Cbval = f[i * 2 + 1].props["PlaneMean"] * 255 # type: ignore
            computestate(Cdiff, Cbval)

        if omode != 4:
            return preproc[current]
        else:
            text = f'{min(-Cbp1, Cbc0) if Cbc0 > 0 and Cbp1 < 0 else 0.0}{" -> BLEND!!" if min(-Cbp1, Cbc0) >= bthresh else " "}'
            return core.text.Text(clip, text)

    # process
    blendclip = input if dclip is None else dclip
    blendclip = mvf.GetPlane(blendclip, 0)
    if fnr:
        blendclip = core.median.TemporalMedian(blendclip, radius=1) # type: ignore
    blendclip = core.resize.Bilinear(blendclip, int(blendclip.width * 0.125 / xr) * 8, int(blendclip.height * 0.125 / yr) * 8)

    diff = core.std.MakeDiff(blendclip, blendclip[1:])

    if input.format.sample_type == vs.INTEGER:
        neutral = 1 << (input.format.bits_per_sample - 1)
        neutral2 = 1 << input.format.bits_per_sample
        peak = (1 << input.format.bits_per_sample) - 1
        inv_scale = "" if input.format.bits_per_sample == 8 else f"{255 / peak} *"
        scale = 1 if input.format.bits_per_sample == 8 else peak / 255

    elif input.format.sample_type == vs.FLOAT:
        neutral = 0.5 # type: ignore
        neutral2 = 1
        peak = 1
        inv_scale = f"{255 / peak} *"
        scale = peak / 255

    x_diff = f"x {neutral} - abs"
    y_diff = f"y {neutral} - abs"
    xy_diff = f"x y + {neutral2} - abs"

    expr = (f"{x_diff} {y_diff} < {x_diff} {xy_diff} < and "                       # ((abs(x-128) < abs(y-128)) && (abs(x-128) < abs(x+y-256)) ?
            f"{x_diff} {inv_scale} dup sqrt - dup * "                              #  sqrt(abs(x-128) - sqrt(abs(x-128)))^2 :
            f"{y_diff} {xy_diff} < "                                               #  abs(y-128) < abs(x+y-256) ?
            f"{y_diff} {inv_scale} dup sqrt - dup * "                              #  sqrt(abs(y-128) - sqrt(abs(y-128)))^2 :
            f"{xy_diff} {inv_scale} dup sqrt - dup * ? ? "                         #  sqrt(abs(x+y-256) - sqrt(abs(x+y-256)))^2
            f"x {neutral} - y {neutral} - * 0 > {-scale} {scale} ? * {neutral} +") # ) * ((x-128) * (y-128) > 0 ? -1 : 1) + 128

    mask = core.std.Expr([diff, diff[1:]], [expr])

    if omode > 1:
        # LumaDifference(blendclip.trim(1,0),blendclip.trim(3,0))
        Cdiff = mvf.PlaneCompare(blendclip[1:], blendclip[3:], mae=True, rmse=False, psnr=False, cov=False, corr=False)

        # AverageLuma(mask.trim(1,0))
        Cbval = mvf.PlaneStatistics(mask[1:], mean=True, mad=False, var=False, std=False, rms=False)
    else:
        # LumaDifference(blendclip,blendclip.trim(2,0)
        Cdiff = mvf.PlaneCompare(blendclip, blendclip[2:], mae=True, rmse=False, psnr=False, cov=False, corr=False)

        # AverageLuma(mask)
        Cbval = mvf.PlaneStatistics(mask, mean=True, mad=False, var=False, std=False, rms=False)

    prop_src = []
    for i in range(preroll, 0, -1):
        prop_src.append(Cdiff[0] * i + Cdiff)
        prop_src.append(Cbval[0] * i + Cbval)
    prop_src.extend([Cdiff, Cbval]);

    last = core.std.FrameEval(input, functools.partial(evaluate, clip=input, core=core), prop_src=prop_src)

    recl = haf_ChangeFPS(haf_ChangeFPS(last, last.fps_num * 2, last.fps_den * 2), last.fps_num, last.fps_den)

    return recl

[theChaosCoder]

@myrsloik Thx works great!