Encoder and decoder support of large image size

[wantehchang]

This issue tracks the work to allow the libavif encoder and decoder to support larger image size.

Note that it requires an AV1 codec that supports large frame size. For example, this is libaom's issue on encoder support of large frame size: https://aomedia.issues.chromium.org/issues/353371270

[wantehchang]

@jzern @vrabaud @y-guyon

Here is a representative example of (unsigned) integer overflow, taken from avifpng.c. Note that rgb.rowBytes is of the uint32_t type.

     for (uint32_t y = 0; y < rgb.height; ++y) {
         rowPointers[y] = &rgb.pixels[y * rgb.rowBytes];
     }

There are three ways to fix this.

1) Add a size_t cast to y or rgb.rowBytes:

     for (uint32_t y = 0; y < rgb.height; ++y) {
         rowPointers[y] = &rgb.pixels[(size_t)y * rgb.rowBytes];
     }

2) Declare the loop index variable y as size_t:

     for (size_t y = 0; y < rgb.height; ++y) {
         rowPointers[y] = &rgb.pixels[y * rgb.rowBytes];
     }

3) Replace the multiplication with addition of a pointer variable with rgb.rowBytes:

     uint8_t * rgbRow = rgb.pixels;
     for (uint32_t y = 0; y < rgb.height; ++y) {
         rowPointers[y] = rgbRow;
         rgbRow += rgb.rowBytes;
     }

Which solution do you prefer?

I prefer solution 3, because it is also a form of optimization ("strength reduction"). It requires more changes.

[y-guyon]

I prefer solution 3 too.