search

NVIDIA / libdeflate

Heavily optimized library for DEFLATE/zlib/gzip and NVIDIA GDEFLATE compression and decompression
Other
13 stars 2 forks source link

libdeflate_gdeflate_compress_bound sometimes returns values that are too small, resulting in memory corruption #1

Open elasota opened 4 months ago

elasota commented 4 months ago

While running some tests, GDeflateDemo in the DirectStorage demo crashed due to stack corruption. Upon testing, it turns out that this was due to libdeflate_gdeflate_compress writing more data than libdeflate_gdeflate_compress_bound indicated was the upper limit.

Please see the attached file and extract "gdeflatecrash.bin" from it. Calling libdeflate_gdeflate_compress_bound with a 65536 page size returns 65708 bytes but calling libdeflate_gdeflate_compress on the contents of gdeflatecrash.bin will output 65736 bytes.

This is with the compression level set to 12 (i.e. same as in DirectStorage GDeflateDemo).

gdeflatecrash.zip

elasota commented 4 months ago

This seems to actually be actually a 2-part bug: libdeflate_gdeflate_compress_bound returns too small of a value, but libdeflate_gdeflate_compress is also supposed to fail safely if more output is written than is specified in the page buffer, which it sometimes does, but with this sample, it is writing memory past the end of the buffer.