Waifu2x-cunet brightness bug

[dnjulek]

The current cunet-model version makes the output darker than it should be.

It is a known bug in VS's Waifu2x ports, as it happens in the caffe port and was introduced to ncnn-vulkan in R4. ncnn-vulkan-R3.2 is the only one that doesn't have the bug.

Here a comparison showing the input and the difference between a spline resize, vs-mlrt-v2 and vs-mlrt-v8, apparently the bug was introduced after v2: https://slow.pics/c/myFP5fXB

[WolframRhodium]

Could you provide your script for verification? The parameters in these models are exactly the same.

[dnjulek]

ncnn-vulkan-R3.2 and ncnn-vulkan-R4 also use the same model file, the problem may be in the implementation.

I already tried to use core.akarin.Expr(rgbs, "x 0 1 clamp"), but the result was the same

#vs-mlrt-v8
from vsmlrt import Waifu2x, Waifu2xModel, Backend
import vapoursynth as vs
core = vs.core

src = core.imwri.Read("sample.png")
rgbs = src.resize.Point(format=vs.RGBS)
spline = rgbs.resize.Spline36(1920, 1080)
spline_y = spline.resize.Point(format=vs.YUV444PS, matrix=1).std.ShufflePlanes([0], vs.GRAY)

w2x = Waifu2x(rgbs, noise=-1, scale=2, model=Waifu2xModel.cunet,
              backend=Backend.ORT_CUDA())

w2x_y = w2x.resize.Point(format=vs.YUV444PS, matrix=1).std.ShufflePlanes([0], vs.GRAY)

w2x_y.set_output()

########################################
#vs-mlrt-v2
import vapoursynth as vs
core = vs.core

src = core.imwri.Read("sample.png")
rgbs = src.resize.Point(format=vs.RGBS)
spline = rgbs.resize.Spline36(1920, 1080)
spline_y = spline.resize.Point(format=vs.YUV444PS, matrix=1).std.ShufflePlanes([0], vs.GRAY)

w2x = rgbs.ort.Model(r"C:\Program Files\VapourSynth\plugins\waifu2x\cunet\noise0_scale2.0x_model.onnx")
w2x_y = w2x.resize.Point(format=vs.YUV444PS, matrix=1).std.ShufflePlanes([0], vs.GRAY)

w2x_y.set_output()

[WolframRhodium]

noise should be 0 in vs-mlrt-v8.

https://github.com/AmusementClub/vs-mlrt/blob/a2358339acd8a88c00c98dbcc558d15415117373/scripts/vsmlrt.py#L210-L213

[dnjulek]

My bad, I used the wrong model in v2, my issue is with noise disabled. Testing scale2.0x_model.onnx now in v2 the result was the same as v8, so the problem might be in the model. noise0_scale2.0x_model.onnx = correct brightness scale2.0x_model.onnx = darker output

[WolframRhodium]

Yeah, I use the same model from waifu2x-caffe so the issue is similar.

[dnjulek]

ncnn-vulkan-R3.2 is the only one without this problem for noise disabled, the problem is that it is very slow, and the bug was added in the last release (R4), even though very few changes were made between this versions.

[WolframRhodium]

Could you verify whether the problem is related to tilesize? It seems r3.2 used tilesize=180, whereas currently vs-mlrt sets it to the video dimensions.

[dnjulek]

The result was the same: https://slow.pics/c/hczpYTtr We are discussing this in IEW discord server, feel free to join: https://discord.gg/qxTxVJGtst

[WolframRhodium]

Personally, I believe there is not guarantee that the average brightness remains constant after filtered by waifu2x. Such requirement can be satisfied by post-processing.