search

CarVac / librtprocess

A project to make RawTherapee's processing algorithms more readily available.
Other
51 stars 23 forks source link

Demoisaiced values are out of range #55

Open Vincent-FA opened 4 years ago

Vincent-FA commented 4 years ago

I'm passing an array of values [0, 65535] to vng4_demosaic and I'm getting values between -1632.537354 and 86103.359375 as a result. What's happening, what should I do?

With rcd_demosaic, I'm getting proper values.

heckflosse commented 4 years ago

We had this issue in RT as well. Last year I made some changes to RT demosaickers to avoid negative values. This changes are not yet in librtprocess. For now you could just check for negative values and clip them to zero. It's not only the case for vng4, also for amaze, where negative values occur sometimes at red/blue locations. I already searched the amaze code for the culprit though without success :(

Vincent-FA commented 4 years ago

What about the fact that values are also above 65535, should we clip them as well? A third above seems like losing a lot of information. Thanks

heckflosse commented 4 years ago

In RT we don't clip above 65535, only below zero. RCD implementation in librtprocess clamps to [0;65535]. Not sure what's the best. Though we should try to get consistent behaviour for the demosaickers.

Vincent-FA commented 4 years ago

What do you do with the values above if you don't clip them?

darix commented 4 years ago

@heckflosse which changes from RT need to be pulled into librtprocess?