vpxenc fails coefficient range checking when encoding a simple source

[GoogleCodeExporter]

libvpx was configured the following way:
$ ./configure --enable-vp9-highbitdepth --disable-multithread 
--enable-coefficient-range-checking --enable-debug

The attached Y4M file can't be encoded by vpxenc:

$ vpxenc allegrodvt-overflow.y4m --passes=1 --min-q=30 -o /dev/null
Pass 1/1 frame    1/0          0B    2855 us 350.26 fps [ETA  unknown] vpxenc: 
../vp9/common/vp9_idct.h:92: check_range: Assertion `input <= (32767)' failed.
[1]    20262 abort      vpxenc allegrodvt-overflow.y4m --passes=1 --min-q=30 -o 
/dev/null

The video source is a 1-frame 64x64 monochrome Y4M, showing a 2x1 checkers-like 
pattern (full white, full black).
As a clumsy workaround, the encoder might be recompiled with coefficient range 
checks disabled.
The generated VP9 bitstream would not be compliant, and the way to decode it 
would be implementation defined.

The issue arises in the intermediate values of the IDCT.
AFAICT, in libvpx, the SSE implementation of the IDCT clips the values, whereas 
the C implementation overflows (rollover).
Why not make the clipping behaviour normative? Overflowing doesn't make a lot 
of sense IMHO.

Original issue reported on code.google.com by sebastien.alaiwan@allegrodvt.com on 29 Jun 2015 at 3:50

Attachments:

• allegrodvt-overflow.y4m

[GoogleCodeExporter]

Making clipping behavior normative is not possible since that would mean 
changing the bitstream specification, which has been frozen since summer 2013. 

Encoders encountering setup and input YUV such as your example are free to 
handle this in whatever appropriate manner. For example, the overflow could be 
detected and trigger a re-encode with different settings (lossless, different 
QP, etc.).

Original comment by pkapsenb@google.com on 30 Jun 2015 at 3:59

[GoogleCodeExporter]

Original comment by ya...@google.com on 30 Jun 2015 at 5:21

• Changed state: Available

[GoogleCodeExporter]

Looking closely at the issue, it is a 16x16 block residual values of uniform 
-255. The dequantized coefficients are: 

ffff8d2b ffffd968 ffffe8d8 ffffef60 fffff2f0 fffff550 fffff680 fffff7b0 
fffff848 fffff8e0 fffff978 fffffa10 fffffa10 fffffa10 fffffaa8 fffffaa8
       0        0        0  all (0)s. 

Because the transform type is DCT_ADST, so the inverse transform used is 
iadst_idct combination. The iadst can produce values out of range in stage 3.

A possible fix is to encode side specific inverse transforms with range check. 

Original comment by ya...@google.com on 30 Jun 2015 at 9:48

[GoogleCodeExporter]

Original comment by ya...@google.com on 30 Jun 2015 at 10:00

[GoogleCodeExporter]

Also in this particular example, configuring without --enable-vp9-highbitdepth 
enables the encoder to produce a valid vp9 output because the mode that led to 
out-of-range coefficient in evaluation is not selected in actual encoding.

Original comment by ya...@google.com on 30 Jun 2015 at 11:52

Attachments:

• xx.webm