Updates start_decoder's calculation of f->temp_memory_required to more accurately reflect the required size of the part_classdata array in decode_residue.
In decode_residue, the value actual_size is derived from the variable n, which is sourced from the variable n2 in vorbis_decode_packet_rest. n2 is equal to one of the blocksizes divided by 2 (of which f->blocksize_1 will always be the greater value). For residue type 2, decode_residue multiplies its copy of n by 2 (back to the original blocksize).
The faulty bounding in start_decoder unconditionally divides f->blocksize_1 by 2, even for residue type 2. This patch corrects the bounding to use the full f->blocksize_1 for residue type 2.
This bug may have implications for anything using the alloc buffer feature, but I don't have an input that would cause this (the setup requires a much larger temp buffer than decode_residue). I found this bug via a heap corruption crash while fuzzing libxmp, which uses a work buffer derived from f->temp_memory_required instead of alloca. The alloca doesn't use f->temp_memory_required, so it does NOT have this bug.
Updates
start_decoder's calculation off->temp_memory_requiredto more accurately reflect the required size of thepart_classdataarray indecode_residue.In
decode_residue, the valueactual_sizeis derived from the variablen, which is sourced from the variablen2invorbis_decode_packet_rest.n2is equal to one of the blocksizes divided by 2 (of whichf->blocksize_1will always be the greater value). For residue type 2,decode_residuemultiplies its copy ofnby 2 (back to the original blocksize).The faulty bounding in
start_decoderunconditionally dividesf->blocksize_1by 2, even for residue type 2. This patch corrects the bounding to use the fullf->blocksize_1for residue type 2.This bug may have implications for anything using the alloc buffer feature, but I don't have an input that would cause this (the setup requires a much larger temp buffer than
decode_residue). I found this bug via a heap corruption crash while fuzzing libxmp, which uses a work buffer derived fromf->temp_memory_requiredinstead ofalloca. Theallocadoesn't usef->temp_memory_required, so it does NOT have this bug.