search

bitemyapp / brotli2-rs

Brotli encoders/decoers for Rust
Apache License 2.0
28 stars 13 forks source link

New attempt to update to latest upstream brotli sys #31

Open dekellum opened 4 years ago

dekellum commented 4 years ago

Attempting to resurrect #25 and use https://github.com/google/brotli/releases/tag/v1.0.7

dekellum commented 4 years ago

There might be other issues as well (Systest?) but I can reliably reproduce the problem found for macos on CI (on linux in my case):

     Running `target/release/examples/all-read-write-roundtrips`
thread 'main' panicked at 'assertion failed: avail_in != data.len() || status == DeStatus::Finished', src/write.rs:282:9

So this example appears to be used as a sort of integration test. It includes an array of 6 datas which are round-trip tested. The last, largest one fails. It is 10MiB of the 44 byte:

     let v2 = vec![44; 10 * 1024 * 1024];

Stepping through this in gdb, a return for this case on decompress is:

brotli_sys::BROTLI_DECODER_RESULT_NEEDS_MORE_OUTPUT => Ok(DeStatus::NeedOutput)

But BrotliDecoder::write (write.rs:L262) does not specifically handle this case. If I compare the BrotliDecoderDecompressStream (decode.c) between the prior v0.6.0 and v1.0.7, this return code and associate invariant (below) also existed in v0.6.0, but this same test case apparently doesn't trip it. 

/*  Invariant: input stream is never overconsumed:
[...]
    - when result is "needs more output" decoder MUST ensure that it doesn't
      hold more than 7 bits in bit reader; this saves client from swapping input
      buffer ahead of time
[...]
*/

It might suggest that the approach of the writer to use "Zero-length output buf to keep it all inside the decompresser buffer" is not entirely sound? I'm not at present sure how to continue trying to fix this.

#0  <brotli2::write::BrotliDecoder<W> as std::io::Write>::write (self=0x7fffffffdbc8, data=&[u8](len: 12) = {...})
    at /home/david/src/brotli2-rs/src/write.rs:276
#1  0x0000555555567df8 in std::io::Write::write_all (self=0x7fffffffdbc8, buf=&[u8](len: 26) = {...})
    at /rustc/f43c34a134358471726f25fe5973b8c7e177c825/src/libstd/io/mod.rs:1356
#2  0x000055555556112d in all_read_write_roundtrips::main::iowritedecode (data=&[u8](len: 26) = {...})
    at examples/all-read-write-roundtrips.rs:57
#3  0x000055555555e58f in all_read_write_roundtrips::main::{{closure}} (data=&[u8](len: 10485760) = {...})
    at examples/all-read-write-roundtrips.rs:68
#4  0x0000555555560735 in all_read_write_roundtrips::main () at examples/all-read-write-roundtrips.rs:84
alexcrichton commented 4 years ago

Ah sorry I'm definitely not that well versed in the brotli native library nor the intricacies here, I'd be blind debugging basically trying to help out, and unfortunately I don't have a ton of time to dig in right now.

(tbh I'm probably not a great maintainer of this crate, I made it as a one-off thing long ago and haven't really touched it much since then, if you or others would like to take over maintaining it I'd be happy to transfer!)

dekellum commented 4 years ago

Thanks, I hope you don't mind me keeping this open while I'm trying to find a fix! If I get anywhere and continue using it, I'll definitely request maintenance permissions.