Incorrectly decoding deflate as gzip

[esheri3]

First off, thank you for taking the time to build reqwest.

Does reqwest support the parsing of chunked responses? If so, do you have an example you could point me to? I am running into StringError(CRC32) panics when calling io::copy / response::copy_to on a chunked response. Thanks in advance.

[seanmonstar]

Yes, reqwest will decode chunked encoded bodies automatically. The string error sounds like something specific to the destination writer (crc32?). For instance, is it assuming utf-8 bytes, and is the response encoded in something else?

[esheri3]

thank you for the quick reply. im trying to follow the documentation re: "reqwest::Response::copy_to" as closely as possible but i am still getting the error. below is the sample code, the http response code / headers and the panic (redacted where appropriate). im on the following version in Cargo.toml: "reqwest = "^0.8.1""

-- code -- info!("downloading archive: [redacted]/file.zip"); let mut response = reqwest::get(&url).unwrap();

info!("response code: {}", response.status()); for header in response.headers().iter() { info!("headers: {:?}", header); }

let mut buf: Vec = vec![]; let _ = response.copy_to(&mut buf).expect("copy_to failed"); // fails here

-- response -- response code: 200 OK headers: Date: Thu, 30 Nov 2017 14:17:01 GMT headers: Server: Apache headers: Last-Modified: Wed, 29 Nov 2017 16:40:51 GMT headers: ETag: "2b80147-1117fa-55f21cc6bc557-gzip" headers: Accept-Ranges: bytes headers: Vary: Accept-Encoding,User-Agent headers: Transfer-Encoding: chunked headers: Content-Type: application/zip

-- panic -- thread 'main' panicked at 'copy_to failed: Error { kind: Io(Error { repr: Custom(Custom { kind: InvalidData, error: StringError("CRC32 mismatched: value=1493567468, expected=2178660496") }) }), url: None }', src/libcore/result.rs:906:4 stack backtrace: 0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace 1: std::sys_common::backtrace::_print 2: std::panicking::default_hook::{{closure}} 3: std::panicking::default_hook 4: std::panicking::rust_panic_with_hook 5: std::panicking::begin_panic 6: std::panicking::begin_panic_fmt 7: rust_begin_unwind 8: core::panicking::panic_fmt 9: core::result::unwrap_failed 10: <core::result::Result<T, E>>::expect 11: [redacted]::install::sdk 12: [redacted]::run 13: [redacted]::main 14: __rust_maybe_catch_panic 15: std::rt::lang_start 16: main

[esheri3]

after looking at decoder.rs, i suspect the issue im experiencing is because there is no decoder for "zip" files. it looks as though reqwest either supports gzip via "Decoder::gzip" otherwise plain text via "Decoder::plain_text". im guessing plain_text is getting triggered & expects utf8 which might explain the CRC32 errors im seeing.

let me know if you see anything differently. im going to explore my ability to migrate to a gzip file format instead of zip as a temporary work-around.

[esheri3]

update: im experiencing the same CRC32 error with the same code even if the file type is .tar.gz ("Content-Type: application/z-gzip", "Transfer-Encoding: chunked").

[esheri3]

creating the client using the following api fixed the original issue for me. apparently the request needed the deflate value in the acceptencoding request header. unless you feel there is an easier way to invoke the client api, i believe this "issue" to be resolved. thanks for the guidance.

-- code --

        let mut client = reqwest::Client::builder()
            .gzip(true)
            .build().expect("build failed");

        let mut response = client.get(&url)
            .header(header::AcceptEncoding(vec![header::qitem(reqwest::header::Encoding::Deflate)]))
            .send().expect("send failed");

[seanmonstar]

Ah, OK, so was the issue that the response was deflate encoded, and reqwest was trying to decode it a gunzip?

[esheri3]

Yes, that is correct. Is deflate encoding something that could be detected / supported automatically similar to gzip?

[seanmonstar]

Quite probably, though I don't have the time at the moment to look too much into it myself.

[crowlandsimms]

Hi @seanmonstar. I hope you don't mind me commenting - I stumbled across this while looking for something else, and I think this issue can be closed. I believe you added support for deflate in v0.11.4?