vorbis_analysis_wrote increments v->pcm_current by vals, and this incremented value can be used by _preextrapolate_helper right after to allocate a float array in the stack v->pcm_current positions large. Clearly, since alloca does not check that there is enough stack space available to satisfy the allocation request, this can lead to a stack overflow and memory corruption, which at best have no effect, more likely cause segmentation faults, and at worst introduce security risks.
The documentation for vorbis_analysis_buffer and vorbis_analysis_wrote does not specify a maximum value for vals. It states that "1024 is a reasonable choice", but callers are free to use larger or smaller counts as they wish. Therefore, libvorbis not handling this case is undesirable behavior.
To better handle this case without throwing the performance benefits of alloca out the window, let's check whether the allocation would exceed 256 KiB (an estimate for the minimum stack space available is 1 MiB, which is the default on Windows platforms), and if so fall back to a heap allocated array. The heap array that may be allocated for this purpose is freed when vorbis_dsp_clear is called. _preextrapolate_helper takes neglible execution time compared to the encoding process for usual sample block sizes, though.
vorbis_analysis_wroteincrementsv->pcm_currentbyvals, and this incremented value can be used by_preextrapolate_helperright after to allocate a float array in the stackv->pcm_currentpositions large. Clearly, sinceallocadoes not check that there is enough stack space available to satisfy the allocation request, this can lead to a stack overflow and memory corruption, which at best have no effect, more likely cause segmentation faults, and at worst introduce security risks.The documentation for
vorbis_analysis_bufferandvorbis_analysis_wrotedoes not specify a maximum value forvals. It states that "1024 is a reasonable choice", but callers are free to use larger or smaller counts as they wish. Therefore,libvorbisnot handling this case is undesirable behavior.To better handle this case without throwing the performance benefits of
allocaout the window, let's check whether the allocation would exceed 256 KiB (an estimate for the minimum stack space available is 1 MiB, which is the default on Windows platforms), and if so fall back to a heap allocated array. The heap array that may be allocated for this purpose is freed whenvorbis_dsp_clearis called._preextrapolate_helpertakes neglible execution time compared to the encoding process for usual sample block sizes, though.