rust-lang / flate2-rs

DEFLATE, gzip, and zlib bindings for Rust
https://docs.rs/flate2
Apache License 2.0
895 stars 161 forks source link

flate2::Decompress::decompress consumes more input than actually used #434

Closed link2xt closed 6 hours ago

link2xt commented 14 hours ago

flate2::Decompress::decompress is documented as "consuming only as much input as needed and writing as much output as possible". However in my tests it consumes the whole input even if output buffer size is not sufficient. Because of this I cannot use flate2 to decode the IMAP stream incrementally using async-compression crate: https://github.com/async-email/async-imap/pull/112

Here is a failing test where output buffer is only 8 bytes long, but the whole input is consumed:

#[test]
fn deflate_decoder_partial() {
    // Decompresses to
    // "* QUOTAROOT INBOX \"User quota\"\r\n* QUOTA \"User quota\" (STORAGE 76 307200)\r\nA0001 OK Getquotaroot completed (0.001 + 0.000 secs).\r\n"
    let input = vec![
        210, 82, 8, 12, 245, 15, 113, 12, 242, 247, 15, 81, 240, 244, 115, 242, 143, 80, 80, 10,
        45, 78, 45, 82, 40, 44, 205, 47, 73, 84, 226, 229, 210, 130, 200, 163, 136, 42, 104, 4,
        135, 248, 7, 57, 186, 187, 42, 152, 155, 41, 24, 27, 152, 27, 25, 24, 104, 242, 114, 57,
        26, 24, 24, 24, 42, 248, 123, 43, 184, 167, 150, 128, 213, 21, 229, 231, 151, 40, 36, 231,
        231, 22, 228, 164, 150, 164, 166, 40, 104, 24, 232, 129, 20, 104, 43, 128, 104, 3, 133,
        226, 212, 228, 98, 77, 61, 94, 46, 0, 0, 0, 0, 255, 255,
    ];

    // Create very small output buffer.
    let mut output = vec![0; 8];

    let zlib_header = false;
    let mut decompress = flate2::Decompress::new(zlib_header);

    let flush_decompress = flate2::FlushDecompress::None;
    let status = decompress.decompress(&input, &mut output, flush_decompress).unwrap();
    assert_eq!(status, flate2::Status::Ok);

    // Should not consume everything, there is not enough space in the buffer for the output.
    assert_ne!(decompress.total_in(), input.len() as u64);
}

Here miniz_oxide::inflate::stream::inflate returns bytes_consumed equal to the whole input size even though there was not enough space in the output buffer: https://github.com/rust-lang/flate2-rs/blob/1a28821dc116dac14178858be056e4a58ef8f501/src/ffi/rust.rs#L73

I thought about reporting the bug directly to miniz_oxide, but don't see in its documentation any guarantee that it will consume only as much as needed.

link2xt commented 14 hours ago

The test passes with --no-default-features --features zlib, so probably should be upstreamed to miniz_oxide.

link2xt commented 12 hours ago

I upstreamed the issue to miniz_oxide: https://github.com/Frommi/miniz_oxide/issues/158