Feature request: Implement exclusion mask for unprocessed areas in based_aa

[kgrabs]

Eedi3 antialiasing with the source passed as sclip causes a large amount of edge areas to trigger the cutoff for interpolation direction change penalizing, especially with the aggressive default gamma and vthresh settings in based_aa. This results in many lines with steep or pure horizontal/vertical slope to be upscaled and then downscaled with no effect from eedi3 and then merged with edge masking. A simple exclusion mask made from a binary expression for each dimension is able to prevent such processing, although I recommend making it a non-default toggle for lack of proper testing.

What follows is a stripped down version of based_aa taken from old prototypes for reference:

def based_aa(clip):
    #completed edge mask ready for merge
    mask = clip.std.Prewitt().std.Binarize(70<<8).std.Maximum().std.BoxBlur(0,1,1,1,1)

    #transpose and double
    supersampled_y = clip.std.Transpose()
    supersampled_y = waifu2x(supersampled_y)

    # first aa pass and check horizontal interpolation area of effect 
    antialiased_y = eedi3s(supersampled_y, sclip=supersampled_y)
    processing_area_y = core.std.Expr([supersampled_y, antialiased_y], 'x y = 0 65535 ?').std.Transpose()

    # second aa pass and add vertical aoe to horizontal aoe
    supersampled_x = antialiased_y.std.Transpose()
    antialiased_x = eedi3s(supersampled_x, sclip=antialiased_y)
    processing_area = core.std.Expr([supersampled_x, antialiased_x, processing_area_y], 'x y = z 65535 ?')

    # resample processing mask to output resolution, expand, and remove unaffected areas from edge mask
    processing_area = resize_mclip(processing_area, clip.width, clip.height).std.Binarize(1).std.Maximum().std.Maximum().std.Maximum()
    mask = core.std.Expr([mask, processing_area], 'x y min')

    final = antialiased_x.resize.Bicubic(clip.width, clip.height)

    return clip.std.MaskedMerge(final, mask)