Memory stomp due to libdeflate_gdeflate_compress_bound returning too small of a value and libdeflate_gdeflate_compress ignoring the output limit

microsoft / DirectStorage

DirectStorage for Windows is an API that allows game developers to unlock the full potential of high speed NVMe drives for loading game assets.

MIT License

694 stars 79 forks source link

Memory stomp due to libdeflate_gdeflate_compress_bound returning too small of a value and libdeflate_gdeflate_compress ignoring the output limit #41

Open elasota opened 3 months ago

elasota commented 3 months ago

While running some tests, GDeflateDemo crashed due to stack corruption. Upon testing, it turns out that this was due to libdeflate_gdeflate_compress writing more data than libdeflate_gdeflate_compress_bound indicated was the upper limit.

Reproduction steps:

Extract "gdeflatecrash.bin" from the attached file
Compile GDeflateDemo in debug mode so stack corruption checks are enabled
Run GDeflateDemo /compress gdeflatecrash.bin .

gdeflatecrash.zip

elasota commented 3 months ago

I guess this is actually a 2-part bug: libdeflate_gdeflate_compress_bound returns too small of a value, but libdeflate_gdeflate_compress is also supposed to fail safely if more output is written than is specified in the page buffer, which it sometimes does, but with this sample, it is stomping memory above the provided limit.