search

ericmckean / webm

Automatically exported from code.google.com/p/webm
0 stars 0 forks source link

Use of unitialized value in vp9 loop filter #637

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
This only happens when vp9 is built without assembly:
./configure --disable-sse --disable-sse2 --disable-ssse3 --disable-sse4_1 
--disable-mmx --disable-sse3 --enable-debug --enable-debug-libs

./test_libvpx --gtest_filter=VP9/TestVectorTest.MD5Match/101

Looks like it overreads memory past the end of the frame buffer.

WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f657e61cfa6 in clamp ./vp9/common/vp9_common.h:51
    #1 0x7f657e61cfa6 in signed_char_clamp vp9/common/vp9_loopfilter_filters.c:17
    #2 0x7f657e61cfa6 in filter4 vp9/common/vp9_loopfilter_filters.c:83
    #3 0x7f657e61cfa6 in vp9_loop_filter_horizontal_edge_c vp9/common/vp9_loopfilter_filters.c:119
    #4 0x7f657e61acfb in filter_selectively_horiz vp9/common/vp9_loopfilter.c:423
    #5 0x7f657e618ae0 in filter_block_plane vp9/common/vp9_loopfilter.c:982
    #6 0x7f657e618ae0 in vp9_loop_filter_rows vp9/common/vp9_loopfilter.c:1024
    #7 0x7f657e718348 in decode_tile vp9/decoder/vp9_decodframe.c:651
    #8 0x7f657e711614 in decode_tiles vp9/decoder/vp9_decodframe.c:748
    #9 0x7f657e711614 in vp9_decode_frame vp9/decoder/vp9_decodframe.c:1010
    #10 0x7f657e559a2f in vp9_receive_compressed_data vp9/decoder/vp9_onyxd_if.c:335
    #11 0x7f657e552851 in decode_one vp9/vp9_dx_iface.c:349
    #12 0x7f657e552851 in vp9_decode vp9/vp9_dx_iface.c:443
    #13 0x7f657e383f94 in vpx_codec_decode vpx/src/vpx_decoder.c:127
    #14 0x7f657e1bf045 in libvpx_test::DecoderTest::RunLoop(libvpx_test::CompressedVideoSource*) (test_libvpx+0x1d40
45)
    #15 0x7f657e2eaaef in (anonymous namespace)::TestVectorTest_MD5Match_Test::TestBody() (test_libvpx+0x2ffaef)
    #16 0x7f657e2eca7c in non-virtual thunk to (anonymous namespace)::TestVectorTest_MD5Match_Test::TestBody() test/
test_vector_test.cc:214
    #17 0x7f657e803789 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/src/src/gtest.cc:2090
    #18 0x7f657e79e228 in testing::Test::Run() third_party/googletest/src/src/gtest.cc:2161
    #19 0x7f657e7a49d7 in testing::TestInfo::Run() third_party/googletest/src/src/gtest.cc:2338
    #20 0x7f657e7a7459 in testing::TestCase::Run() third_party/googletest/src/src/gtest.cc:2445
    #21 0x7f657e7cc8ab in testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/src/gtest.cc:4237
    #22 0x7f657e8083c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/src/gtest.cc:2090
    #23 0x7f657e7cac59 in testing::UnitTest::Run() third_party/googletest/src/src/gtest.cc:3871
    #24 0x7f657e2dfba4 in main (test_libvpx+0x2f4ba4)
    #25 0x7f657cb5e76c in __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226
    #26 0x7f657e0be900 in _start (test_libvpx+0xd3900)
  Uninitialized value was created by a heap allocation
    #0 0x7f657e08c6e3 in malloc (test_libvpx+0xa16e3)
    #1 0x7f657e38bb86 in vpx_memalign vpx_mem/vpx_mem.c:125
    #2 0x7f657e55edd7 in vp9_realloc_frame_buffer vpx_scale/generic/yv12config.c:152
    #3 0x7f657e55f68c in vp9_alloc_frame_buffer vpx_scale/generic/yv12config.c:206
    #4 0x7f657e6029c7 in vp9_alloc_frame_buffers vp9/common/vp9_alloccommon.c:100
    #5 0x7f657e72409e in apply_frame_size vp9/decoder/vp9_decodframe.c:531
    #6 0x7f657e722898 in setup_frame_size vp9/decoder/vp9_decodframe.c:561
    #7 0x7f657e70b9be in read_uncompressed_header vp9/decoder/vp9_decodframe.c:862
    #8 0x7f657e70b9be in vp9_decode_frame vp9/decoder/vp9_decodframe.c:968
    #9 0x7f657e559a2f in vp9_receive_compressed_data vp9/decoder/vp9_onyxd_if.c:335
    #10 0x7f657e552851 in decode_one vp9/vp9_dx_iface.c:349
    #11 0x7f657e552851 in vp9_decode vp9/vp9_dx_iface.c:443
    #12 0x7f657e383f94 in vpx_codec_decode vpx/src/vpx_decoder.c:127
    #13 0x7f657e1bf045 in libvpx_test::DecoderTest::RunLoop(libvpx_test::CompressedVideoSource*) (test_libvpx+0x1d4045)
    #14 0x7f657e2eaaef in (anonymous namespace)::TestVectorTest_MD5Match_Test::TestBody() (test_libvpx+0x2ffaef)
    #15 0x7f657e2eca7c in non-virtual thunk to (anonymous namespace)::TestVectorTest_MD5Match_Test::TestBody() test/test_vector_test.cc:214
    #16 0x7f657e803789 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/src/src/gtest.cc:2090
    #17 0x7f657e79e228 in testing::Test::Run() third_party/googletest/src/src/gtest.cc:2161
    #18 0x7f657e7a49d7 in testing::TestInfo::Run() third_party/googletest/src/src/gtest.cc:2338
    #19 0x7f657e7a7459 in testing::TestCase::Run() third_party/googletest/src/src/gtest.cc:2445
    #20 0x7f657e7cc8ab in testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/src/src/gtest.cc:4237
    #21 0x7f657e8083c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/src/src/gtest.cc:2090

The same with Valgrind:

==2932== Conditional jump or move depends on uninitialised value(s)
==2932==    at 0x5E99C0: vp9_loop_filter_horizontal_edge_c 
(vp9/common/vp9_loopfilter_filters.c:69)
==2932==    by 0x5E6F5A: filter_selectively_horiz 
(vp9/common/vp9_loopfilter.c:423)
==2932==    by 0x5E743C: filter_block_plane (vp9/common/vp9_loopfilter.c:982)
==2932==    by 0x5E8F3E: vp9_loop_filter_rows (vp9/common/vp9_loopfilter.c:1024)
==2932==    by 0x62D5C8: decode_tiles (vp9/decoder/vp9_decodframe.c:748)
==2932==    by 0x633D4E: vp9_decode_frame (vp9/decoder/vp9_decodframe.c:1010)
==2932==    by 0x5AEE8B: vp9_receive_compressed_data 
(vp9/decoder/vp9_onyxd_if.c:335)
==2932==    by 0x5AD479: vp9_decode (vp9/vp9_dx_iface.c:349)
==2932==    by 0x501CD5: vpx_codec_decode (vpx/src/vpx_decoder.c:127)
==2932==    by 0x4662D7: 
libvpx_test::DecoderTest::RunLoop(libvpx_test::CompressedVideoSource*) 
(test/decode_test_driver.cc:22)
==2932==    by 0x4C82AB: (anonymous 
namespace)::TestVectorTest_MD5Match_Test::TestBody() 
(test/test_vector_test.cc:231)
==2932==    by 0x66531C: void 
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, 
void>(testing::Test*, void (testing::Test::*)(), char const*) 
(third_party/googletest/src/src/gtest.cc:2090)

Original issue reported on code.google.com by euge...@chromium.org on 3 Oct 2013 at 9:44

GoogleCodeExporter commented 9 years ago

Original comment by ya...@google.com on 17 Oct 2013 at 10:37