VP9 inverse transform may cause overflow assertion

[GoogleCodeExporter]

It is possible for an encoder to generate a bitstream that contains a 
coefficient block that causes an assertion during inverse transform. 
In such cases, what should the proper handling be of the overflow? Saturate or 
increase precision?

The dequantized coefficient block triggering the issue is the following 16x16 
(base_qindex was 247):
int coeffs[256] = 
{14820  ,3134   ,0  ,0  ,0  ,0  ,1567   ,1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,1567,
        1567    ,1567   ,0  ,0  ,1567   ,0  ,1567   ,1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        -3134   ,-1567  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        -6268   ,-1567  ,0  ,0  ,-1567  ,0  ,-1567  ,-1567  ,0  ,0  ,0  ,0  ,0  ,0  ,0, -1567,
        -3134   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        -1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        0   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        -1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        1567    ,1567   ,1567   ,1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        -3134   ,-1567  ,-1567  ,-1567  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        7835    ,3134   ,1567   ,1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        0   ,0  ,0  ,0  ,0  ,0  ,0, 0   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        -1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0,
        4701    ,1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,1567   ,0  ,0  ,0  ,0  ,0, 0,
        0   ,0  ,0  ,0  ,0  ,-1567  ,-1567  ,-1567  ,-1567  ,-1567  ,0, 0   ,0  ,0  ,0  ,0,
        -1567   ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0  ,0};
When performing idct horizontal followed by iadst vertical, an overflow occurs 
in stage 3 of the vertical transform.

Original issue reported on code.google.com by pieter.k...@gmail.com on 15 Nov 2013 at 9:08

[GoogleCodeExporter]

Original comment by ya...@google.com on 18 Nov 2013 at 7:01

• Changed state: Assigned

[GoogleCodeExporter]

Take out the assertion check from inverse transforms. If the transform 
coefficients were constructed to cause intermediate steps of inverse transform 
overflow, the codec will just let it overflow without breaking the decoding 
flow. The rate-distortion optimization loop at encoder will make this fall back 
to other coding modes (usually of smaller prediction/transform size).

https://gerrit.chromium.org/gerrit/#/c/67842/

Original comment by jingn...@google.com on 22 Nov 2013 at 11:30

• Changed state: Fixed

[GoogleCodeExporter]

Not all VP9 encoders will be able to handle the issue this way...

Original comment by pieter.k...@gmail.com on 22 Nov 2013 at 11:57