[vp9] out of bounds read in temporal_filter_iterate_c

[GoogleCodeExporter]

v1.3.0-2571-g56186c2

Given the similarity it's possible this bug exists in vp8 too. The fix should 
include a unit test which exercises all versions of this function similar to 
fdct, idct, variance, etc.

$ ./configure --target=generic-gnu --enable-debug
$ valgrind ./test_libvpx \
    --gtest_filter=VP9/BordersTest.TestEncodeHighBitrate/0
==20158== Memcheck, a memory error detector
==20158== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==20158== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==20158== Command: ./test_libvpx 
--gtest_filter=VP9/BordersTest.TestEncodeHighBitrate/0
==20158== 
Note: Google Test filter = VP9/BordersTest.TestEncodeHighBitrate/0
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from VP9/BordersTest
[ RUN      ] VP9/BordersTest.TestEncodeHighBitrate/0
==20158== Invalid read of size 4
==20158==    at 0x5DE9D5: temporal_filter_iterate_c (vp9_temporal_filter.c:307)
==20158==    by 0x5DF2CF: vp9_temporal_filter_prepare 
(vp9_temporal_filter.c:441)
==20158==    by 0x59DD7A: vp9_get_compressed_data (vp9_encoder.c:2539)
==20158==    by 0x58D4A4: encoder_encode (vp9_cx_iface.c:805)
==20158==    by 0x5440ED: vpx_codec_encode (vpx_encoder.c:227)
==20158==    by 0x4853A7: 
libvpx_test::Encoder::EncodeFrameInternal(libvpx_test::VideoSource const&, 
unsigned long) (encode_test_driver.cc:62)
==20158==    by 0x486DFB: 
libvpx_test::EncoderTest::RunLoop(libvpx_test::VideoSource*) 
(encode_test_driver.cc:22)
==20158==    by 0x440ADB: (anonymous 
namespace)::BordersTest_TestEncodeHighBitrate_Test::TestBody() 
(borders_test.cc:62)
==20158==    by 0x68B74C: void 
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, 
void>(testing::Test*, void (testing::Test::*)(), char const*) 
(gtest-all.cc:3562)
==20158==    by 0x68326F: testing::Test::Run() (gtest-all.cc:3635)
==20158==    by 0x683537: testing::TestInfo::Run() (gtest-all.cc:3810)
==20158==    by 0x6836F0: testing::TestCase::Run() (gtest-all.cc:3928)
==20158==  Address 0x9af8fc is not stack'd, malloc'd or (recently) free'd
==20158== 
==20158== 
==20158== Process terminating with default action of signal 11 (SIGSEGV): 
dumping core
==20158==  Access not within mapped region at address 0x9AF8FC
==20158==    at 0x5DE9D5: temporal_filter_iterate_c (vp9_temporal_filter.c:307)
==20158==    by 0x5DF2CF: vp9_temporal_filter_prepare 
(vp9_temporal_filter.c:441)
==20158==    by 0x59DD7A: vp9_get_compressed_data (vp9_encoder.c:2539)
==20158==    by 0x58D4A4: encoder_encode (vp9_cx_iface.c:805)
==20158==    by 0x5440ED: vpx_codec_encode (vpx_encoder.c:227)
==20158==    by 0x4853A7: 
libvpx_test::Encoder::EncodeFrameInternal(libvpx_test::VideoSource const&, 
unsigned long) (encode_test_driver.cc:62)
==20158==    by 0x486DFB: 
libvpx_test::EncoderTest::RunLoop(libvpx_test::VideoSource*) 
(encode_test_driver.cc:22)
==20158==    by 0x440ADB: (anonymous 
namespace)::BordersTest_TestEncodeHighBitrate_Test::TestBody() 
(borders_test.cc:62)
==20158==    by 0x68B74C: void 
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, 
void>(testing::Test*, void (testing::Test::*)(), char const*) 
(gtest-all.cc:3562)
==20158==    by 0x68326F: testing::Test::Run() (gtest-all.cc:3635)
==20158==    by 0x683537: testing::TestInfo::Run() (gtest-all.cc:3810)
==20158==    by 0x6836F0: testing::TestCase::Run() (gtest-all.cc:3928)
==20158==  If you believe this happened as a result of a stack
==20158==  overflow in your program's main thread (unlikely but
==20158==  possible), you can try to increase the size of the
==20158==  main thread stack using the --main-stacksize= flag.
==20158==  The main thread stack size used in this run was 8388608.
==20158== 
==20158== HEAP SUMMARY:
==20158==     in use at exit: 32,166,742 bytes in 26,510 blocks
==20158==   total heap usage: 69,845 allocs, 43,335 frees, 57,441,390 bytes 
allocated
==20158== 
==20158== LEAK SUMMARY:
==20158==    definitely lost: 0 bytes in 0 blocks
==20158==    indirectly lost: 0 bytes in 0 blocks
==20158==      possibly lost: 31,490,162 bytes in 22,268 blocks
==20158==    still reachable: 676,580 bytes in 4,242 blocks
==20158==         suppressed: 0 bytes in 0 blocks
==20158== Rerun with --leak-check=full to see details of leaked memory
==20158== 
==20158== For counts of detected and suppressed errors, rerun with: -v
==20158== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
Segmentation fault

Original issue reported on code.google.com by jz...@google.com on 3 May 2014 at 6:00

[GoogleCodeExporter]

Fixed here:
https://gerrit.chromium.org/gerrit/#/c/69978/

Original comment by agra...@google.com on 6 May 2014 at 11:56

• Changed state: Fixed

[GoogleCodeExporter]

This still requires a test.

Original comment by jz...@google.com on 7 May 2014 at 3:06

• Changed state: Started

[GoogleCodeExporter]

ping

Original comment by jz...@google.com on 27 May 2014 at 10:09