deflate::Decoder always decompresses whole blocks (seems inefficient, unexpected and even unsecure)

[radekm]

It seems that deflate::Decoder always decompresses whole block.

If that is true it means:

• Reading a single byte may consume huge amounts of memory. If the block is huge then it may even run out of memory before the first byte is returned to the user.
• If this code gets into a web server or some security related software then reading certain archives could destabilize the whole program.
• I think that the behavior when reading 1 byte could consume unlimited amount of memory and take unlimited time is surprising to users of io::Read which is implemented by deflate::Decoder.

Here is an example. Reading 1 byte from 10 MB stream consumes 1.3 GB of memory:

    let mut buf = Vec::new();

    let mut writer = BitWriter::new(Cursor::new(&mut buf));
    writer.write_bits(1, 0);  // Not last block
    writer.write_bits(2, 1);  // Fixed Huffman
    writer.write_bits(8, 0b10001110);  // Literal A

    for _ in 0..6000_000 {
        writer.write_bits(8, 0b10100011);  // Max length
        writer.write_bits(5, 0);  // Distance 1
    }
    // End block.
    writer.write_bits(7, 0);
    writer.flush();

    println!("Written {} compressed bytes", buf.len());

    let mut dec = Decoder::new(buf.as_slice());
    let mut out = [1u8];
    dec.read_exact(&mut out);

[radekm]

This behavior also makes it harder to read corrupted archives (even if you are interested in some prefix which is ok).

[radekm]

This is completely wrong. I was using libflate instead of flate2.

And flate2 behaves correctly.