search

Nullus157 / async-compression

Adaptors between compression crates and Rust's async IO types
https://docs.rs/async-compression
Apache License 2.0
392 stars 77 forks source link

panic: cannot consume from pending buffer #298

Open Turbo87 opened 3 days ago

Turbo87 commented 3 days ago

In https://github.com/rust-lang/crates.io/pull/9622 we upgraded crates.io to the latest async-compression release (v0.4.14), which appears to be causing issues on our staging environment:

https://rust-lang.sentry.io/share/issue/9e273e28600f48a5aec2ad9e127afbbf/

The code inside the object_store crate is throwing a "cannot consume from pending buffer" panic, but the root cause seems to be originating from the async-compression crate. We use both crates roughly like this:

let reader = object_store::buffered::BufReader::new(store, &meta);
let decompressor = Decompressor::from_extension(reader, path.extension())?;

I assume that https://github.com/Nullus157/async-compression/pull/294 is causing this issue, but I don't know the code well enough to find the exact bug in the code changes that are causing this behavior.

Turbo87 commented 2 days ago

@robjtede it looks like #299 didn't fix the problem, and GitHub won't let me reopen this issue 😅

NobodyXu commented 2 days ago

@Turbo87 Which compression algorithm are you using?

I suspect the decoder impl is wrong.

Turbo87 commented 1 day ago

Which compression algorithm are you using?

zstd and gzip (see https://github.com/rust-lang/crates.io/blob/51c3f84456ecc1fc6131a1f82478536e05e4569d/crates/crates_io_cdn_logs/src/compression.rs). I just checked, we're seeing the issue for both of them apparently.

NobodyXu commented 1 day ago

Thanks, I will have to take a look at them.

Surely I have missed something.