Mismatch decoded output for C and SSE2 code base

[GoogleCodeExporter]

The decoder output is different between
../../configure --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3 
--disable-sse4_1 --disable-avx --disable-avx2 

and 
../../configure. 

Quick debugging indicates there is mismatch between iadst16 sse2 and c code. 

Original issue reported on code.google.com by ya...@google.com on 29 Sep 2014 at 6:44

Attachments:

• broken_idct_stream.vp9

[GoogleCodeExporter]

Also, configure with --enable-coefficient-range-checking --enable-debug 
It indicate there is an out-of-range issue happening in intermediate stage of 
inverse transforms, i.e., the stream itself is an invalid(non-conforming) vp9 
stream. 

Original comment by ya...@google.com on 29 Sep 2014 at 6:45

[GoogleCodeExporter]

Original comment by ya...@google.com on 29 Sep 2014 at 7:19

• Changed state: Available

[GoogleCodeExporter]

Original comment by jingn...@google.com on 29 Sep 2014 at 7:26

• Changed state: Started

[GoogleCodeExporter]

The reported bit-stream was found to have overflow issue in the intermediate 
stage of inverse transform (16x16 inverse ADST, stage 3) when truncating from 
32-bit to 16-bit. It is an invalid bit-stream. The behavoir of handling 
intermediate stage overflow is not defined. The current SSE2 and ARM 
implementations take saturation for overflow cases, while the C code retains 
the lower 16-bit.

Original comment by jingn...@google.com on 30 Sep 2014 at 11:01

[GoogleCodeExporter]

Original comment by ya...@google.com on 30 Sep 2014 at 11:41

• Changed state: WontFix