transfer-encoding:chunked

[12089897411]

decode_response cannot decode the chunked response body

[omjadas]

It is my understanding that hyper's Body should handle chunking under the hood. Could you provide some more details regarding the exact issue you are seeing?

[12089897411]

This is an example, when a response has both “content-encoding: gzip” and “transfer-encoding: chunked”, use decode_response to decode it, an invalid gzip header will occur

[tokio::test]

async fn test() { let author = webpki_roots::TLS_SERVER_ROOTS.0; let mut data1 = vec![]; let owned = author.iter().map(|f| { OwnedTrustAnchor::from_subject_spki_name_constraints(f.subject, f.spki, f.name_constraints) }); let mut root = RootCertStore::empty(); root.add_server_trust_anchors(owned); let data ="GET / HTTP/1.1 Host: www.baidu.com Cookie: BIDUPSID=03D61FA7F976C8AF8545DEB15009D1B6; PSTM=1675992990; BAIDUID=03D61FA7F976C8AF995FDE16D982CACA:FG=1; BD_UPN=12314753; BAIDUID_BFESS=03D61FA7F976C8AF995FDE16D982CACA:FG=1; ZFY=F9G:BsOHg3qgVoAYnIvBgHJrkXi0y6LkfdLSduvu62Ck:C; baikeVisitId=706fbd35-3985-498c-a125-5d263315d2e7 Sec-Ch-Ua: \"Not:A-Brand\";v=\"99\", \"Chromium\";v=\"112\" Sec-Ch-Ua-Mobile: ?0 Sec-Ch-Ua-Platform: \"Windows\" Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.138 Safari/537.36 Sec-Purpose: prefetch;prerender Purpose: prefetch Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7 Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1 Sec-Fetch-Dest: document Accept-Encoding: gzip, deflate Accept-Language: zh-CN,zh;q=0.9 Connection: close

".as_bytes();

let config = ClientConfig::builder()
    .with_safe_defaults()
    .with_root_certificates(root)
    .with_no_client_auth();
let connector = TlsConnector::from(Arc::new(config));
let tcp_stream = TcpStream::connect(("www.baidu.com", 443)).await;
if let Ok(domain) = ServerName::try_from("www.baidu.com") {
    if let Ok(mut tls_stream) = connector.connect(domain, tcp_stream.ok().unwrap()).await {
        tls_stream.write_all(data).await;
        tls_stream.read_to_end(&mut data1).await;
    }
}
let mut headers = [httparse::EMPTY_HEADER; 100];
let mut header_map = HeaderMap::new();

let mut response = httparse::Response::new(&mut headers);

let mut resp = Response::default();
let mut body = vec![];
if let Ok(size) = response.parse(&data1) {
    for item in response.headers {
        if let Ok(name) = HeaderName::from_bytes(item.name.as_bytes()) {
            if let Ok(value) = HeaderValue::from_bytes(item.value) {
                header_map.insert(name, value);
            };
        }
    }
    body = data[size.unwrap()..].to_vec();
}
resp = Response::new(Body::from(body));
*resp.status_mut() = StatusCode::from_u16(response.code.unwrap()).ok().unwrap();
*resp.version_mut() = match response.version {
    Some(0) => Version::HTTP_10,

    Some(1) => Version::HTTP_11,

    _ => unreachable!(),
};
*resp.headers_mut() = header_map;
let mut result = decode_response(resp).unwrap();
println!(
    "{:?}",
    String::from_utf8_lossy(&to_bytes(result.body_mut()).await.unwrap())
);

}

mistake:: thread 'widget::http_repeater_viewer::test' panicked at 'called Result::unwrap() on an Err value: hyper::Error(Body, Custom { kind: InvalidData, error: "Invalid gzip header" })

I solved it by referencing the chunked_transfer library, if possible, I hope hudsucker can handle it

[12089897411]

The response body of this response will have an extra cc7 in the header, so it is an invalid gzip header.It bothered me for many days, and finally I compromised and used the chunked-transfer library.

[omjadas]

The issue in your example is that you are not constructing the Body correctly. If you have to manually construct the response you will need to combine the chunks correctly (with the chunk lengths and \r\ns stripped). Instead of doing that, however, I would recommend using a hyper Client to make the HTTP request, as it will handle a lot of things for you including constructing a correct Body. If you just need to make HTTP requests and are not using any of the other functionality of hudsucker, I would recommend something more high-level such as reqwest.