search

kleopatra999 / webm

Automatically exported from code.google.com/p/webm
0 stars 0 forks source link

Motion compensation filters too many rows when scaling (causing segmentation fault) #853

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
In vp9/common/vp9_convolve.c the function convolve computes intermediate_height 
via the calculation:

    int intermediate_height = (((h - 1) * y_step_q4 + 15) >> 4) + SUBPEL_TAPS;
    if (intermediate_height < h)
        intermediate_height = h;

I believe this calculation should be:

    int intermediate_height = (((h - 1) * y_step_q4 + y0_q4) >> 4) + SUBPEL_TAPS;

I believe the current code has two problems:

    1. More rows than are needed are filtered, this makes the codec slower than necessary
    2. When scaling is being used, the increase of intermediate_height can result in the access of illegal memory locations, causing a segmentation fault.  (Suppose a 64x64 block is being predicted from a smaller reference frame.  It should only need a few rows of the reference frame, but the current code increases the number of rows to 64 and causes accesses beyond the range of allocated memory.)

Original issue reported on code.google.com by peter.de...@gmail.com on 4 Sep 2014 at 8:39

GoogleCodeExporter commented 9 years ago 
Fix for this issue:
https://gerrit.chromium.org/gerrit/#/c/71718/

has been merged.

Original comment by ya...@google.com on 3 Oct 2014 at 5:39