search

Asd-g / avs-mlrt

ML Runtimes for AviSynth+.
GNU General Public License v3.0
19 stars 0 forks source link

Update mlrt.avsi wrapper to fix models' output #4

Closed DonCanjas closed 1 year ago

DonCanjas commented 1 year ago

The current process with mlrt_w2x() will introduce a tint. I also am aware that when using, at least, cunet model will damage borders in a extension of 4px.

Here's an example of a w2x wrapper that fixes all its problems for, at least, cunet model: https://github.com/Irrational-Encoding-Wizardry/vs-scale/blob/master/vsscale/scale.py#L289-L338

So basically the process is:

By the way, I'm not noticing any tinting when converting a yuv clip to rgbps, but am noticing a luminosity change when extracting luma and packing it in RGB. So I'm assuming you have the luminosity problem fixed. Also not sure if you have the padding already.

ColorBarsHD()
ExtractY()
z_PointResize(width(last)*2, Height(last)*2)
ColorBarsHD()
ExtractY().ConvertBits(32)

MergeRGB(last, last, last)

mlrt_W2x(noise=-1, scale=2, tiles=4, overlap_w=16)

ExtractR()
ColorBarsHD()
ExtractY().ConvertBits(32)

MergeRGB(last, last, last)
Expr(last, "x 0 1 clip")

mlrt_W2x(noise=-1, scale=2, tiles=4, overlap_w=16)

ExtractR()
Expr(last, "x 0.5 255 / +")

Limiting the RGBPS input would also benefit when using DPIR.

LSMASHVideoSource("test.mp4")
LSMASHVideoSource("test.mp4")

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="470bg:601:170m:l=>RGB:601:170m:f")

mlrt_DPIR(strength=90, model=3, tiles=6, overlap_w=32)
LSMASHVideoSource("test.mp4")

z_ConvertFormat(pixel_type="RGBPS", colorspace_op="470bg:601:170m:l=>RGB:601:170m:f")

Expr("x 0 1 clip")

mlrt_DPIR(strength=90, model=3, tiles=6, overlap_w=32)
Asd-g commented 1 year ago

You can try the updated version - added mlrt_W2x padding and input clamping.

Expr("x 0.5 255 / +") is not perfect either.

ColorBarsHD()
ExtractY()
z_ConvertFormat(pixel_type="rgbps")
Expr(last, "x 0 1 clip")

mlrt_W2x(noise=-1, scale=2, tiles=4, overlap_w=16)

Expr(last, "x 0.5 255 / +")

The output of above script is a bit reddish compared to the following

ColorBarsHD()
ExtractY()
z_ConvertFormat(pixel_type="rgbps")
z_PointResize(width(last)*2, Height(last)*2)

I agree with this comment. Such operation should be part of the post processing/subjective preferences.

Btw...

ColorBarsHD()
ExtractY().ConvertBits(32)

MergeRGB(last, last, last)

CombinePlanes is faster than MergeRGB here:

ColorBarsHD()
ExtractY().ConvertBits(32)

#MergeRGB(last, last, last)
CombinePlanes(last, planes="rgb", source_planes="yyy", pixel_type="rgbps")

Or you can just use the fastest method:

ColorBarsHD()
ExtractY()
z_ConvertFormat(pixel_type="rgbps")
DonCanjas commented 1 year ago

I'm not sure if it's really necessary to use both addborders and fillborders when a simple mirror of the edges with z_PointResize works. And the script should make sure the res is kept at mod4 after padding, so adding 4px borders might be too simplistic in cases where the padded resolution isn't mod4.

Also, Expr("x 0.5 255 / +") is for Y/YUV while Expr(last, "x 0 1 clip") is for RGB. Maybe I didn't explain myself properly. Another problem would be if converting to 32 bit before converting to rgb, the limiter will clamp the image unnecessarily making it look faded. That could cause a problem when rescaling video or taking a Y32 input.

Thanks for the tip with CombinePlanes, didn't think of that. Tho seems like to me it's a bit faster to use CombinePlanes then converting with avsresize, while using a bit more RAM. So maybe its speed might depend on how much ram is available.

Asd-g commented 1 year ago

I'm not sure if it's really necessary to use both addborders and fillborders when a simple mirror of the edges with z_PointResize works. And the script should make sure the res is kept at mod4 after padding, so adding 4px borders might be too simplistic in cases where the padded resolution isn't mod4.

AddBorders+FillBorders is faster than z_PointResize with identical output. If the input resolution is mod4, AddBorders(4,4,4,4) keeps mod4 too. If the input isn't mod4, the resolution after AddBorders(4,4,4,4) is again non-mod4. The script already checks if the resolution after padding is mod4 - https://github.com/Asd-g/avs-mlrt/blob/main/mlrt.avsi#L158

Also, Expr("x 0.5 255 / +") is for Y/YUV while Expr(last, "x 0 1 clip") is for RGB. Maybe I didn't explain myself properly. Another problem would be if converting to 32 bit before converting to rgb, the limiter will clamp the image unnecessarily making it look faded. That could cause a problem when rescaling video or taking a Y32 input.

I'm sorry but I cannot understand what you're saying. Can you show some examples to explain what you mean?

DonCanjas commented 1 year ago

The script already checks if the resolution after padding is mod4

Alright, that's good.

I'm sorry but I cannot understand what you're saying. Can you show some examples to explain what you mean?

Sorry, I was typing that late and got confused myself. Expr("x 0.5 255 / +") is supposed to correct tint/luminosity, but as I said in my first comment, I didn't really notice a tint. You're right, I can see a redish tint after running Expr("x 0.5 255 / +").

What I was trying to say is that using ConvertBits(32).z_ConvertFormat(pixel_type="RGBPS", ... ) will look faded. But I noticed that by using ConvertBits(32, fulls=false, fulld=true).z_ConvertFormat(pixel_type="RGBPS", ... ) it won't look faded, so seems like everything is alright.

Seems to me that everything is alrigth

Asd-g commented 1 year ago

There is indeed minor tint introduced but as I wrote Expr("x 0.5 255 / +") isn't perfect solution. Currently everyone can use whatever he prefers (add Expr("x 0.5 255 / +") or not).

What I was trying to say is that using ConvertBits(32).z_ConvertFormat(pixel_type="RGBPS", ... ) will look faded.

Ah, such type of faded. As you figured out always use ConvertBits(32, fulls=false, fulld=true) for converting Y/YUV to 32-bit.