stack overflow in vp9_sub_pixel_variance16x16_sse2

[GoogleCodeExporter]

Since upgrading to libvpx 1.3.0 in the OpenBSD ports tree and thus introducing 
VP9 support I have had a report from a user of a stack overflow being exposed 
on OpenBSD/i386. It would appear to be in the SSE2 code. It sounds as if 
essentially encoding anything as a source to VP9 results in this crash.

I am going to see about getting access to an i386 system to see if I can 
reproduce the issue and if so build with debug symbols to try and provide a 
more complete backtrace.

Means of exposing the issue..

ffmpeg -y \
    -i input.mp4 \
    -c:v libvpx-vp9 \
    -c:a libopus \
    -strict experimental \
    output.webm

(gdb) bt
#0  0x079bd3e1 in kill () at <stdin>:2
#1  0x079f94c1 in __stack_smash_handler (func=0x2db944c0 
"vp9_sub_pixel_variance16x16_sse2", damaged=1760) at 
/usr/src/lib/libc/sys/stack_protector.c:61
#2  0x0dc6589e in vp9_sub_pixel_variance16x16_sse2 () from 
/usr/local/lib/libvpx.so.5.0
#3  0x0dc2c467 in vp9_find_best_sub_pixel_tree () from 
/usr/local/lib/libvpx.so.5.0
#4  0x0dc489c9 in vp9_rd_pick_inter_mode_sb () from /usr/local/lib/libvpx.so.5.0
#5  0x0dc1cb98 in pick_sb_modes () from /usr/local/lib/libvpx.so.5.0
#6  0x0dc1a14c in rd_pick_partition () from /usr/local/lib/libvpx.so.5.0
#7  0x0dc1a868 in rd_pick_partition () from /usr/local/lib/libvpx.so.5.0
#8  0x0dc1a868 in rd_pick_partition () from /usr/local/lib/libvpx.so.5.0
#9  0x0dc16136 in encode_frame_internal () from /usr/local/lib/libvpx.so.5.0
#10 0x0dc147f0 in vp9_encode_frame () from /usr/local/lib/libvpx.so.5.0
#11 0x0dc40107 in encode_frame_to_data_rate () from /usr/local/lib/libvpx.so.5.0
#12 0x0dc3ee17 in vp9_get_compressed_data () from /usr/local/lib/libvpx.so.5.0
#13 0x0dc06895 in vp9e_encode () from /usr/local/lib/libvpx.so.5.0
#14 0x0db81fa3 in vpx_codec_encode () from /usr/local/lib/libvpx.so.5.0
#15 0x0e307342 in vp8_encode () from /usr/local/lib/libavcodec.so.20.0

Original issue reported on code.google.com by brad.ope...@gmail.com on 10 Jun 2014 at 3:20

• Blocked on: #924

[GoogleCodeExporter]

Original comment by jz...@google.com on 23 Jan 2015 at 4:08

• Now blocked on: #924

[GoogleCodeExporter]

As small request. Would be possible to reflect in ./configure's output, that 
use-x86inc was enabled or disabled? For example here it was enabled:

Configuring selected codecs
  enabling vp8_encoder
  enabling vp8_decoder
  enabling vp9_encoder
  enabling vp9_decoder
Configuring for target 'x86-linux-gcc'
  enabling x86
  enabling pic
  enabling runtime_cpu_detect
  enabling mmx
  enabling sse
  enabling sse2
  enabling sse3
  enabling ssse3
  enabling sse4_1
  enabling avx
  enabling avx2
  using yasm
  enabling postproc
  enabling webm_io
  enabling libyuv
Creating makefiles for x86-linux-gcc libs
Creating makefiles for x86-linux-gcc docs

Then I've looked at generated files in source directory and I was sure that 
configure option actually worked.

Original comment by mikolaj....@gmail.com on 28 Jan 2015 at 2:52

[GoogleCodeExporter]

> As small request. Would be possible to reflect in ./configure's output, that
> use-x86inc was enabled or disabled? For example here it was enabled:
> [...]

Seems reasonable. I opened #936 to track.

[1] https://code.google.com/p/webm/issues/detail?id=936

Original comment by jz...@google.com on 29 Jan 2015 at 4:27

[GoogleCodeExporter]

Original comment by fgalli...@google.com on 30 Jan 2015 at 8:37

• Added labels: Pri-1

[GoogleCodeExporter]

Issue 924 was resolved.

https://code.google.com/p/webm/issues/detail?id=924

Original comment by yunqingw...@google.com on 30 Jan 2015 at 8:41

[GoogleCodeExporter]

I'm compiling libvpx commit be6aeada and I see "#define CONFIG_USE_X86INC 0" 
after configure. Do you think can --disable-use-x86inc as default for OpenBSD 
be reverted?

Original comment by mikolaj....@gmail.com on 31 Jan 2015 at 9:23

[GoogleCodeExporter]

This has been merged.
3be948d Merge "configure: enable x86inc for all intel platforms"
769fe6e configure: enable x86inc for all intel platforms

All issues related to this bug should be addressed now. There are still some 
issues with building in general (unit tests with clang++ and bad sse2 
intrinsics causing failure with gcc), but those can go in new bugs.

Thanks again for the report and the help in working through this.

Original comment by jz...@google.com on 5 Feb 2015 at 8:22

• Changed state: Fixed