scaler crash

[GoogleCodeExporter]

webrtc currently has a yuvscaler crash due to a buffer being too small.

If the user allocates too little memory for the UV plane and/or passes bad 
parameters, the yuv scaler can crash because it assumes the width of UV plane 
is (width + 1) / 2
The real issue is the user passing bad parameters, but libyuv could check the 
(width + 1) / 2 is not more than the stride.
stride can be more than width, for alignment, and it may be 0 if the user wants 
to pass 1 row and/or duplicate a row.  But its not normally useful to pass a 
width that is less than stride.
width should normally be respected, but the code could assert, or clamp width.
negative strides are legal, so abs(stride) should be used.

For higher level functions, like ConvertToI420, it may make sense to pass the 
buffer size(s) and compare against stride to ensure pointers wont be used 
outside the buffer.

Original issue reported on code.google.com by fbarch...@google.com on 20 Jun 2012 at 1:41

[GoogleCodeExporter]

This issue appears to be webrtc underallocating buffers.
libyuv will add functions to estimate buffer sizes in the future.

Unrelated, but a scaler malfunction was found with bilinear SSE2.  Its unclear 
what side effects it had, but PSNR was wrong, so SSE2 bilinear is disabled.  
SSSE3 and C remain.

Original comment by fbarch...@google.com on 14 Jul 2012 at 1:17

[GoogleCodeExporter]

The crash is due to webrtc allocating w*h*3/2 bytes.  This is not fixed in 
webrtc yet, but avoided by ensuring images are a multiple of 2.
Marking as 'invalid' as there is no change needed in libyuv.

Original comment by fbarch...@google.com on 21 Jul 2012 at 9:04

• Changed state: Invalid