Incorrect subsampling used in vp9 non420 loopfilter

[GoogleCodeExporter]

In vp9/common/vp9_loopfilter.c: filter_block_plane_non420()

The lines:

912: const int row_step = 1 << ss_x;
913: const int col_step = 1 << ss_y;

Should be:

912: const int row_step = 1 << ss_y;
913: const int col_step = 1 << ss_x;

This causes segmentation faults for certain image sizes and subsampling.

Original issue reported on code.google.com by james.hu...@argondesign.com on 26 Aug 2014 at 2:18

[GoogleCodeExporter]

Original comment by ya...@google.com on 26 Aug 2014 at 7:32

[GoogleCodeExporter]

Original comment by ya...@google.com on 26 Aug 2014 at 7:32

[GoogleCodeExporter]

I see that this is causing filtering to be applied in the wrong places, but I'm 
having trouble reproducing segfaults here. The loop constraints seem to keep it 
in bounds. 

  for (r = 0; r < MI_BLOCK_SIZE && mi_row + r < cm->mi_rows; r += row_step)
    for (c = 0; c < MI_BLOCK_SIZE && mi_col + c < cm->mi_cols; c += col_step)

I've tried various sized 4:2:2 clips in both valgrind and asan and not seen any 
illegal memory operations. 

Can you advise on specific use cases that are causing a segfault? 

Original comment by aconve...@google.com on 28 Aug 2014 at 6:48

[GoogleCodeExporter]

I think the segfault only occurs with 4:4:0 streams. In this case the for loop 
counts double the number of rows and dst->buf is incremented by 8 * stride for 
each row.

I've tried to produce a 4:4:0 stream using libvpx but it doesn't seem to be 
supported.

Is ss_x = 0, ss_y = 1 legal for a VP9 bitstream?

However as you note the loopfilter is being applied in the wrong places for 
4:2:2.

Original comment by james.hu...@argondesign.com on 29 Aug 2014 at 11:44

[GoogleCodeExporter]

Alex, 
Any updates on this issue? 

I've also faced it and confirm segfaults on 4:4:0 streams.

Original comment by ru.xa...@gmail.com on 19 Sep 2014 at 11:26

[GoogleCodeExporter]

I have merged the fix https://gerrit.chromium.org/gerrit/#/c/71647/ by 
Alexander Voronov. 

Much thanks to Alexander.

Original comment by ya...@google.com on 24 Sep 2014 at 4:02

• Changed state: Fixed