AOMediaCodec / libavif

libavif - Library for encoding and decoding .avif files
Other
1.53k stars 195 forks source link

avifImageAllocatePlanes() should use the size_t type for allocation sizes #277

Open wantehchang opened 4 years ago

wantehchang commented 4 years ago

avifImageAllocatePlanes() uses the int type for all local variables, including the allocation sizes fullSize and uvSize. This prevents a grid image to be larger than, say, 64K x 64K on 64-bit platforms.

Specifically, there are three allocation sizes:

    ...
    int fullSize = fullRowBytes * image->height;
    ...
            image->yuvPlanes[AVIF_CHAN_Y] = avifAlloc(fullSize);
    ...
                image->yuvPlanes[AVIF_CHAN_U] = avifAlloc(uvSize);
    ...
                image->yuvPlanes[AVIF_CHAN_V] = avifAlloc(uvSize);
    ...
            image->alphaPlane = avifAlloc(fullRowBytes * image->height);

Note: the allocation of image->alphaPlane could use the local variable fullSize.

The allocation sizes should be calculated using the size_t type instead. In addition, we need to make sure the multiplications do not overflow size_t.

wantehchang commented 4 years ago

The yuvRowBytes and alphaRowBytes members of avifImage should ideally also be declared as size_t, because they are often multiplied with a row index variable. DeclaringyuvRowBytes and alphaRowBytesas size_t ensures those multiplications are done in the size_t type.

But this change will break backward compatibility.