flaccidware / webm

Automatically exported from code.google.com/p/webm
0 stars 0 forks source link

Range check mismatches between 8-bit and high bitdepth versions of IDCT functions #1048

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
Here's the "stage 3" part of the "iadst16" function (8 bit).
  x0 = WRAPLOW(check_range(s0 + s2), 8);
  x1 = WRAPLOW(check_range(s1 + s3), 8);
  x2 = WRAPLOW(check_range(s0 - s2), 8);
  x3 = WRAPLOW(check_range(s1 - s3), 8);
  x4 = WRAPLOW(dct_const_round_shift(s4 + s6), 8);
  x5 = WRAPLOW(dct_const_round_shift(s5 + s7), 8);
  x6 = WRAPLOW(dct_const_round_shift(s4 - s6), 8);
  x7 = WRAPLOW(dct_const_round_shift(s5 - s7), 8);

Here's the "stage 3" part of the "high_bitdepth_iadst16" function.
  x0 = WRAPLOW(s0 + s2, bd);
  x1 = WRAPLOW(s1 + s3, bd);
  x2 = WRAPLOW(s0 - s2, bd);
  x3 = WRAPLOW(s1 - s3, bd);
  x4 = WRAPLOW(highbd_dct_const_round_shift(s4 + s6, bd), bd);
  x5 = WRAPLOW(highbd_dct_const_round_shift(s5 + s7, bd), bd);
  x6 = WRAPLOW(highbd_dct_const_round_shift(s4 - s6, bd), bd);
  x7 = WRAPLOW(highbd_dct_const_round_shift(s5 - s7, bd), bd);

It seems like some explicit calls to "highbd_check_range" are missing. I would 
rather have expected:
  x0 = WRAPLOW(highbd_check_range(s0 + s2, bd), bd);
  x1 = WRAPLOW(highbd_check_range(s1 + s3, bd), bd);
  x2 = WRAPLOW(highbd_check_range(s0 - s2, bd), bd);
  x3 = WRAPLOW(highbd_check_range(s1 - s3, bd), bd);
  x4 = WRAPLOW(highbd_dct_const_round_shift(s4 + s6, bd), bd);
  x5 = WRAPLOW(highbd_dct_const_round_shift(s5 + s7, bd), bd);
  x6 = WRAPLOW(highbd_dct_const_round_shift(s4 - s6, bd), bd);
  x7 = WRAPLOW(highbd_dct_const_round_shift(s5 - s7, bd), bd);

Is this intentional?
Would bitstreams failing these new range checks be considered compliant?
Thanks.

Original issue reported on code.google.com by sebastien.alaiwan@allegrodvt.com on 21 Jul 2015 at 10:18

GoogleCodeExporter commented 8 years ago

Original comment by ya...@google.com on 21 Jul 2015 at 4:08

GoogleCodeExporter commented 8 years ago

Original comment by ya...@google.com on 21 Jul 2015 at 4:08

GoogleCodeExporter commented 8 years ago

Original comment by ya...@google.com on 30 Jul 2015 at 7:28

GoogleCodeExporter commented 8 years ago

Original comment by ya...@google.com on 12 Aug 2015 at 10:06

GoogleCodeExporter commented 8 years ago
We will do better in VP10 with clear definition of range of values in 
transforms.

Original comment by ya...@google.com on 12 Aug 2015 at 10:07