16 bit output is identical to input

[Frechdachs]

If the input clip is 16 bit then the output is identical to the input. 8 bit works fine for me.

Confirmed with the following script:

from vapoursynth import core, YUV420P16

src = core.ffms2.Source("Path/to/some/file.mkv")
src_16 = src.resize.Bicubic(format=YUV420P16)
bilateral = core.bilateralgpu.Bilateral(src_16, sigma_spatial=10, sigma_color=2.25, kernel_size=0, borderMode=4, planes=[0,1,2])
bilateral = core.std.PlaneStats(bilateral, src_16)

for x in bilateral[300:340].frames():
    print(x.props["PlaneStatsDiff"])

This script will always print 0.0.

I'm on Linux by the way.

[WolframRhodium]

The problem arises because current "sigma_color" has not been scaled. That is, if one wants to get a similar result for clip with integer format and bitdepth higher than 8, the "sigma_color" should be multiplied by ((1 << bits_per_sample) - 1) / 255.

I don't normalize it at the beginning because the original OpenCV_CUDA_Bilateral behaves like this. Should I normalize it internally?

[Frechdachs]

Oh, I didn't think of that, sorry.

Should I normalize it internally?

Yes, please. The arguments of most VapourSynth filters are either normalized or in an 8 bit scale and then scaled internally according to input bitdepth, so that's what most users would expect. Anyway, I am going to add this filter as an option to GradFun3, depending on the GPU this can be a very significant speedup.

Two additional things:

#include <opencv2\cudaimgproc.hpp>
#include <vapoursynth\VapourSynth.h>
#include <vapoursynth\VSHelper.h>

1. Do forward slashes in include directives work on Windows, too? If yes, you should use them because compilation on Linux fails otherwise.

2. The description of the two sigmas seems to be switched in README.md.

[WolframRhodium]

Thank you for your advice. Revision has been done.

[Frechdachs]

Thanks, but why is it only normalized if it's an integer? Fine stepping is not really possible since there is a rather big difference between e. g. 2 or 3.

Ah, misread your post again. Thanks, that's exactly what I wanted.

[WolframRhodium]

I don't know how to adjust it for clip in float format properly, so I leave it unchanged.

[Frechdachs]

I don't know how to adjust it for clip in float format properly

Wouldn't division by 255 be enough, or is this incorrect?

Also I looked at the code:

d.sigma_color[i] *= ((1 << d.vi.format->bitsPerSample) - 1) / 255;

The expression on the right is an integer division if I'm not mistaken. That's okay for 16 bit input. But for 10 or 12 bit the calculation will be slightly incorrect.

[WolframRhodium]

Wouldn't division by 255 be enough, or is this incorrect?

You're right. I forgot range conversion in previous test.

But I still don't know how to get same result compared to the core.bilateral.Bilateral(algorithm=2).

[Frechdachs]

But I still don't know how to get same result compared to the core.bilateral.Bilateral(algorithm=2).

Yeah, me neither, but with small sigmaR/sigma_color like it's used in GradFun3, the visible difference is very small. I will test if there even is a visible difference after applying LimitFilter and the detailmask inside GradFun3.

Edit: For example this has a very small visible difference.

bilateral.Bilateral(clip_16, sigmaS=8, sigmaR=2.25/255, algorithm=2, planes=[0,1,2])
bilateralgpu.Bilateral(clip_16, sigma_spatial=8, sigma_color=2.25, kernel_size=0, borderMode=4, planes=[0,1,2])

[WolframRhodium]

I had asked mawen1250 (the author of VapourSynth-Bilateral) this question before, and he confirmed that sigmaR should behave like sigma_color / ValueMax.

I think for small sigmaS/sigma_spatial, difference would be small. When sigmaS is large, his bilateral function uses a trick to improve performance, which may create slight difference.