search

AviSynth / AviSynthPlus

AviSynth with improvements
http://avs-plus.net
967 stars 73 forks source link

out of the range float problem #159

Open realfinder opened 4 years ago

realfinder commented 4 years ago

ColorBars(width=640, height=480, pixel_type="yv12") Convertbits(32) mt_lutxy(last,Invert(),"x x y - 9.9 * +", use_expr=1)

~ ConvertBits(8)

Converttorgb24

it's ok with ConvertBits(8) but with Converttorgb24() it's not

I post this already in doom9, but I make this issue in order to not to be forgotten

pinterf commented 4 years ago

Not a bug. Direct float YUV420->8 bit RGB conversion is done as: 32 bit 4:2:0 input -> 32 bit 4:4:4 -> 32 bit RGB-> 8 bit RGB The issue was causes by having way out-of range YUV values (like magnitude of Y=9 when only 0..1 is allowed), which caused the output RGB values to be garbage as well. (The usual yuv-rgb conversion matrix multiplication cannot cope with such inputs) When forced ConvertBits(8) is used it clamps YUV values to valid ranges before RGB conversion.