stb_vorbis: out-of-memory when allocating f->temp_mults (libFuzzer/ASan).

Describe the bug Certain inputs can cause stb_vorbis to attempt to allocate large amounts of RAM in start_decoder when loading codebook lookup values. There seems to be multiple problems here:

1) codebook_entries is a 24 bit value and codebook_dimensions is a 16 bit value, so the computed codebook_lookup_values can contain values up to 40 bits (this is allowed by the spec). However, this value is stored to a uint32. This isn't necessarily a problem by itself (probably no valid input relies on this), but given certain inputs, this can reach values >2.14b. 2) No attempt is made to verify that this many lookup values can be read from the input stream before allocation. An end-of-packet during any of these read operations is considered an error, so this should probably be done to filter garbage inputs (if possible). 3) codebook_lookup_values is a uint32, but setup_temp_malloc() takes its size parameter as an int (should probably instead be size_t). This casts some large calculated codebook_lookup_values values to negative integers. In setup_temp_malloc(), the aforementioned int sz is then provided to malloc, which takes a size_t parameter. sz is sign extended and then interpreted as an extremely large unsigned value. 4) The replacement malloc for AddressSanitizer emits an error and aborts due to this when allocator_may_return_null=0. This is just a symptom of 2 and 3 though.

To Reproduce Load this test module with stb_vorbis_decode_filename from stb_vorbis built with AddressSanitizer or MemorySanitizer: OGG_lookup_values_oom.zip

Expected behavior stb_vorbis should probably not attempt to allocate several exabytes of memory from an invalid input unless the input file can reasonably contain that many lookup values.

Screenshots

[a@localhost stb]$ ./vorbisA OGG_lookup_values_oom.ogg 
OGG_lookup_values_oom.ogg: =================================================================
==107987==ERROR: AddressSanitizer: requested allocation size 0xffffffffdb900d58 (0xffffffffdb901d58 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
    #0 0x4c33ef in malloc (/home/a/stb/vorbisA+0x4c33ef)
    #1 0x512c91 in setup_temp_malloc(stb_vorbis*, int) /home/a/stb/./stb_vorbis.c:977:11
    #2 0x512c91 in start_decoder(stb_vorbis*) /home/a/stb/./stb_vorbis.c:3863:29
    #3 0x5288ce in stb_vorbis_open_file_section /home/a/stb/./stb_vorbis.c:5060:8
    #4 0x5305b0 in stb_vorbis_open_file /home/a/stb/./stb_vorbis.c:5080:11
    #5 0x5305b0 in stb_vorbis_open_filename /home/a/stb/./stb_vorbis.c:5093:14
    #6 0x5305b0 in stb_vorbis_decode_filename /home/a/stb/./stb_vorbis.c:5350:20
    #7 0x533d34 in main /home/a/stb/vorbisA.cpp:24:17
    #8 0x7fd489841b74 in __libc_start_main (/lib64/libc.so.6+0x27b74)

==107987==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: allocation-size-too-big (/home/a/stb/vorbisA+0x4c33ef) in malloc
==107987==ABORTING

#if 0
clang -O3 -fsanitize=address -fno-omit-frame-pointer -g -Wall -Wextra vorbisA.cpp -ovorbisA
exit
#endif

#include <stdint.h>
#include <stdio.h>
#include "stb_vorbis.c"

int main(int argc, char *argv[])
{
  for(int i = 1; i < argc; i++)
  {
    if(argv[i])
    {
      int16_t *pcm =  NULL;
      int chn, rate, ret;

      fprintf(stderr, "%s: ", argv[i]); fflush(stderr);
      ret = stb_vorbis_decode_filename(argv[i], &chn, &rate, &pcm);
      fprintf(stderr, "%d\n", ret);

      free(pcm);
    }
  }
  return 0;
}

If I find any more of these inputs I'll attach them, but this is the only one that's come up so far.

nothings / stb

stb_vorbis: out-of-memory when allocating f->temp_mults (libFuzzer/ASan). #1248