Core dump calling I420Rotate with irregular size

[GoogleCodeExporter]

What steps will reproduce the problem?
1. set environment variables "LIBYUV_WIDTH=2934" and "LIBYUV_HEIGHT=1798"
2. run "out/Debug/libyuv_unittest --gtest_filter="*I420Rotate*"

What is the expected output? What do you see instead?

According to the documentation (https://code.google.com/p/libyuv/wiki/Rotation) 
this test should pass, but it dumps core.

What version of the product are you using? On what operating system?

I'm running the unit tests from a local gclient build.

Please provide any additional information below.

I'm not sure if this is a valid input, since videos are not commonly found with 
these dimensions.

Original issue reported on code.google.com by byr...@google.com on 6 Jul 2015 at 6:39

[GoogleCodeExporter]

I've also tried dimensions with an aspect ratio of 16:9 at multiples of 8 
units, e.g. 512x288, but it still segfaults. So far the only working inputs for 
I420Rotate that I can find are the two sets of prescribed dimensions: 128x72 
and 1280x720. For all other inputs there have been either omitted values 
(I420TestRotate assert fails), or a buffer overflow (often leading to segfault).

Original comment by byr...@google.com on 6 Jul 2015 at 7:40

[GoogleCodeExporter]

My mistake, the crash is caused by my change to the test driver. I have added 
environment variables LIBYUV_WIDTH_DST and LIBYUV_HEIGHT_DST to set the 
destination dimensions. When these are not an exact multiple of the source 
dimensions, there seem to always be errors. Is it supported to rotate into a 
destination of differing aspect ratio? The API seems to invite this kind of 
usage, but it seems to be invalid input.

Original comment by byr...@google.com on 6 Jul 2015 at 8:05

[GoogleCodeExporter]

I420 function has this API:

int I420Rotate(const uint8* src_y, int src_stride_y,
               const uint8* src_u, int src_stride_u,
               const uint8* src_v, int src_stride_v,
               uint8* dst_y, int dst_stride_y,
               uint8* dst_u, int dst_stride_u,
               uint8* dst_v, int dst_stride_v,
               int src_width, int src_height, enum RotationMode mode);

The original image width and height are specified.  The rotation determines the 
output resolution.
e.g width 1280, height 720 rotated by 90 will be 720x1280 for output resolution.
The destination buffer should normally match that resolution.

Since pointers and strides are used for source and destination buffers, you can 
crop.  But there is no scaling.

Original comment by fbarch...@google.com on 7 Jul 2015 at 9:41

[GoogleCodeExporter]

Thanks for the report.  Looks like its working correctly.

Original comment by fbarch...@google.com on 9 Jul 2015 at 12:59

• Changed state: Invalid