search

iamnpc / libyuv

Automatically exported from code.google.com/p/libyuv
0 stars 0 forks source link

yuv_table overflow on RGBU #6

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
yuv to rgb table uses 2.6 fixed point times 8.0 = 10.6 signed values.

But 2.018 produces a value out of range:
#define RGBU(i) { \
  static_cast<int16>(2.018 * 64 * (i - 128) + 0.5), \
  static_cast<int16>(-0.391 * 64 * (i - 128) + 0.5), \
  0, \
  0 \
}

For RGBU(0xFF)
Since it only affects 2 values, the best solution is to clamp the table entries.

Original issue reported on code.google.com by fbarch...@google.com on 8 Nov 2011 at 7:00

GoogleCodeExporter commented 9 years ago 
The table is fixed.
But in next CL, I'll remove the table method completely.
The SSSE3 version uses multiplies, and the C version mimics the SSSE3 for bit 
exactness

Original comment by fbarch...@chromium.org on 18 Nov 2011 at 9:43