Closed link2xt closed 6 hours ago
flate2::Decompress::decompress is documented as "consuming only as much input as needed and writing as much output as possible". However in my tests it consumes the whole input even if output buffer size is not sufficient. Because of this I cannot use flate2 to decode the IMAP stream incrementally using async-compression crate: https://github.com/async-email/async-imap/pull/112
flate2::Decompress::decompress
flate2
async-compression
Here is a failing test where output buffer is only 8 bytes long, but the whole input is consumed:
#[test] fn deflate_decoder_partial() { // Decompresses to // "* QUOTAROOT INBOX \"User quota\"\r\n* QUOTA \"User quota\" (STORAGE 76 307200)\r\nA0001 OK Getquotaroot completed (0.001 + 0.000 secs).\r\n" let input = vec![ 210, 82, 8, 12, 245, 15, 113, 12, 242, 247, 15, 81, 240, 244, 115, 242, 143, 80, 80, 10, 45, 78, 45, 82, 40, 44, 205, 47, 73, 84, 226, 229, 210, 130, 200, 163, 136, 42, 104, 4, 135, 248, 7, 57, 186, 187, 42, 152, 155, 41, 24, 27, 152, 27, 25, 24, 104, 242, 114, 57, 26, 24, 24, 24, 42, 248, 123, 43, 184, 167, 150, 128, 213, 21, 229, 231, 151, 40, 36, 231, 231, 22, 228, 164, 150, 164, 166, 40, 104, 24, 232, 129, 20, 104, 43, 128, 104, 3, 133, 226, 212, 228, 98, 77, 61, 94, 46, 0, 0, 0, 0, 255, 255, ]; // Create very small output buffer. let mut output = vec![0; 8]; let zlib_header = false; let mut decompress = flate2::Decompress::new(zlib_header); let flush_decompress = flate2::FlushDecompress::None; let status = decompress.decompress(&input, &mut output, flush_decompress).unwrap(); assert_eq!(status, flate2::Status::Ok); // Should not consume everything, there is not enough space in the buffer for the output. assert_ne!(decompress.total_in(), input.len() as u64); }
Here miniz_oxide::inflate::stream::inflate returns bytes_consumed equal to the whole input size even though there was not enough space in the output buffer: https://github.com/rust-lang/flate2-rs/blob/1a28821dc116dac14178858be056e4a58ef8f501/src/ffi/rust.rs#L73
miniz_oxide::inflate::stream::inflate
bytes_consumed
I thought about reporting the bug directly to miniz_oxide, but don't see in its documentation any guarantee that it will consume only as much as needed.
miniz_oxide
The test passes with --no-default-features --features zlib, so probably should be upstreamed to miniz_oxide.
--no-default-features --features zlib
I upstreamed the issue to miniz_oxide: https://github.com/Frommi/miniz_oxide/issues/158
flate2::Decompress::decompress
is documented as "consuming only as much input as needed and writing as much output as possible". However in my tests it consumes the whole input even if output buffer size is not sufficient. Because of this I cannot useflate2
to decode the IMAP stream incrementally usingasync-compression
crate: https://github.com/async-email/async-imap/pull/112Here is a failing test where output buffer is only 8 bytes long, but the whole input is consumed:
Here
miniz_oxide::inflate::stream::inflate
returnsbytes_consumed
equal to the whole input size even though there was not enough space in the output buffer: https://github.com/rust-lang/flate2-rs/blob/1a28821dc116dac14178858be056e4a58ef8f501/src/ffi/rust.rs#L73I thought about reporting the bug directly to
miniz_oxide
, but don't see in its documentation any guarantee that it will consume only as much as needed.