Signed integer overflow in `start_decoder`

[JarLob]

A crafted file may trigger signed integer overflow in c->lookup_values = c->entries * c->dimensions;.

Impact

It is not clear if this is a security issue.

Resources

To reproduce the issue:

1. Make UBSAN build of the following program:

#include "../stb_vorbis.c"
#include <stdint.h>

int main(int argc, char* argv[])
{
    const uint8_t data[] = {0x4f,0x67,0x67,0x53,0x00,0x02,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
                            0x4c,0x69,0x7f,0x64,0x00,0x00,0x00,0x00,0x40,0xae,0xe1,0xfc,0x01,0x1e,
                            0x01,0x76,0x6f,0x72,0x62,0x69,0x73,0x00,0x00,0x00,0x00,0x01,0x44,0xac,
                            0x00,0x00,0xff,0xff,0xff,0xff,0x00,0xf4,0x01,0x00,0xff,0xff,0xff,0xff,
                            0xb8,0x01,0x4f,0x67,0x67,0x53,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
                            0x00,0x00,0x4c,0x69,0x7f,0x64,0x01,0x00,0x00,0x00,0xf1,0x9f,0xfb,0x0f,
                            0x0f,0x5b,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,
                            0xff,0xe8,0x03,0x76,0x6f,0x72,0x62,0x69,0x73,0x2f,0x00,0x00,0x00,0x58,
                            0x69,0x70,0x68,0x2e,0x4f,0x72,0x67,0x20,0x6c,0x69,0x62,0x56,0x6f,0x72,
                            0x62,0x69,0x73,0x20,0x49,0x20,0x32,0x30,0x31,0x34,0x30,0x31,0x32,0x32,
                            0x20,0x28,0x54,0x75,0x72,0x70,0x61,0x6b,0xc3,0xa4,0x72,0xc3,0xa4,0x6a,
                            0x69,0x69,0x6e,0x29,0x01,0x00,0x00,0x00,0x18,0x00,0x00,0x00,0x45,0x4e,
                            0x43,0x4f,0x44,0x45,0x52,0x3d,0x56,0x4c,0x43,0x20,0x6d,0x65,0x64,0x69,
                            0x61,0x20,0x70,0x6c,0x61,0x79,0x65,0x72,0x01,0x05,0x76,0x6f,0x72,0x62,
                            0x69,0x73,0x29,0x42,0x43,0x56,0x01,0x00,0x08,0x00,0x00,0x00,0x31,0x4c,
                            0x20,0xc5,0x80,0xd0,0x90,0x55,0xea,0x28,0xb4,0xd0,0x42,0x0b,0x2d,0xb4,
                            0xd2,0x4a,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x00,0x0e,0x0e,
                            0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,
                            0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e,0x0e};
    size_t size = sizeof(data);

    int chan, samplerate;
    short *output;
    int samples = stb_vorbis_decode_memory(data, size, &chan, &samplerate, &output);
    if (samples >= 0)
        free(output);
    return 0;
}

2. Run the program to hit the error.

/src/stb/tests/../stb_vorbis.c:3860:43: runtime error: signed integer overflow: 738000 * 41897 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /src/stb/tests/../stb_vorbis.c:3860:43 in