search

jungle0755 / libyuv

Automatically exported from code.google.com/p/libyuv
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Signed int overflows in row_gcc.cc #563

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Running webrtc tests under UBSan 
(https://www.chromium.org/developers/testing/undefinedbehaviorsanitizer) 
triggers signed-int overflows in libjingle_media_unittests:

https://build.chromium.org/p/client.webrtc.fyi/builders/Linux%20UBSan/builds/830
/steps/libjingle_media_unittest/logs/stdio

Example:

../../chromium/src/third_party/libyuv/source/row_gcc.cc:2903:25: runtime error: 
signed integer overflow: 128 * 16843009 cannot be represented in type 'int'

Original issue reported on code.google.com by pbos@google.com on 29 Jan 2016 at 10:23

GoogleCodeExporter commented 8 years ago 
[ RUN      ] LibYUVScaleTest.ScaleFrom320x240_Box
../../source/scale.cc:662:28: runtime error: index -2 out of bounds for type 
'int [2]'
filter 3 -      204 us C -       87 us OPT
[       OK ] LibYUVScaleTest.ScaleFrom320x240_Box (1 ms)
[ RUN      ] LibYUVScaleTest.ScaleTo352x288_None

[ RUN      ] LibYUVScaleTest.ScaleDownBy2_None
../../unit_test/../unit_test/unit_test.h:60:33: runtime error: signed integer 
overflow: -629197847 * 214013 cannot be represented in type 'int'
../../unit_test/../unit_test/unit_test.h:60:42: runtime error: signed integer 
overflow: 2145292826 + 2531011 cannot be represented in type 'int'
filter 0 -       12 us C -        1 us OPT
[       OK ] LibYUVScaleTest.ScaleDownBy2_None (1 ms)

Original comment by fbarch...@google.com on 1 Feb 2016 at 8:10

GoogleCodeExporter commented 8 years ago 
LIBYUV_DISABLE_ERMS=1 runyuv

[ RUN      ] LibYUVConvertTest.I400ToI420_Any
../../source/row_gcc.cc:2903:25: runtime error: signed integer overflow: 128 * 
16843009 cannot be represented in type 'int'
[       OK ] LibYUVConvertTest.I400ToI420_Any (1 ms)
[ RUN      ] LibYUVConvertTest.I400ToI420_Unaligned

Original comment by fbarch...@google.com on 1 Feb 2016 at 8:28

GoogleCodeExporter commented 8 years ago 
The following revision refers to this bug:
  https://chromium.googlesource.com/libyuv/libyuv.git/+/9e39c1f27124c0f7c7fb1b88e4012df886e13ca9

commit 9e39c1f27124c0f7c7fb1b88e4012df886e13ca9
Author: Frank Barchard <fbarchard@google.com>
Date: Mon Feb 01 20:29:04 2016

ubsan overflow fix for multiply by 0x01010101

This is an UBSan error reported by libjingle

[ RUN      ] WebRtcVideoFrameTest.ConvertToYUY2BufferStride
[000:000] (videoframe.cc:375): Validate frame passed. format: I420 bpp: 12 
size: 1280x720 bytes: 1382400 expected: 1382400 sample[0..3]: 73, 73, 73, 73
../../chromium/src/third_party/libyuv/source/row_gcc.cc:2903:25: runtime error: 
signed integer overflow: 128 * 16843009 cannot be represented in type 'int'
[8/614] WebRtcVideoFrameTest.ConvertToYUY2BufferStride returned/aborted with 
exit code 1 (32 ms)
[9/614] WebRtcVideoFrameTest.ConvertToYUY2BufferInverted (29 ms)
Note: Google Test filter = WebRtcVideoFrameTest.ConvertToYUY2BufferInverted

The source is uint8 and the multiply is by 0x01010101 to replicate the byte to 
4 bytes.
Changing the constant to 0x01010101u should avoid overflow.

R=harryjin@google.com
TBR=harryjin@google.com
BUG=libyuv:563

Review URL: https://codereview.chromium.org/1657533005 .

[modify] 
http://crrev.com/9e39c1f27124c0f7c7fb1b88e4012df886e13ca9/README.chromium
[modify] 
http://crrev.com/9e39c1f27124c0f7c7fb1b88e4012df886e13ca9/include/libyuv/version
.h
[modify] 
http://crrev.com/9e39c1f27124c0f7c7fb1b88e4012df886e13ca9/source/row_gcc.cc

Original comment by bugdroid1@chromium.org on 1 Feb 2016 at 8:29

GoogleCodeExporter commented 8 years ago 
row_gcc fixed.  unittest and scaler have issues too.

Original comment by fbarch...@google.com on 1 Feb 2016 at 11:19