search

dubhater / vapoursynth-nnedi3

nnedi3 filter for VapourSynth
92 stars 6 forks source link

removed some incorrect floating point clamping #12

Closed IFeelBloated closed 8 years ago

IFeelBloated commented 8 years ago

the main reason you have to do the clamp manually for integers is that you will end up getting the truncated LSB rather than the upper or lower bound of the destination data type when down-casting, and such pain in the a\ does not exist to floating points, every single step of the floating point calculation follows the fixed and precise IEEE 754 standard, and besides, there's just no casting DOWN to floating point here, and, the point of taking floating point over integers as the pixel type is, it gives a flexible range instead of a fixed one like integers that makes saving the mathematically correct but out of range intermediate data possible, and also, some extra precision.

dubhater commented 8 years ago

Thanks. I merged it manually with one small stylistic change.

IFeelBloated commented 8 years ago

do the asm version of the modified functions need modifying as well? sadly I can't do asm...

dubhater commented 8 years ago

I'm pretty sure float pixels were only clamped in those C++ functions.