How to use sparse-http?

[ToBinio]

I am currently trying to use

https://github.com/frewsxcv/rust-crates-index/blob/ff8bebe314cd79b5aa1e5b43221a6bbb1c3560c4/src/sparse_index.rs#L151 https://github.com/frewsxcv/rust-crates-index/blob/ff8bebe314cd79b5aa1e5b43221a6bbb1c3560c4/src/sparse_index.rs#L222

but I don't find a clean way to use them in any crate.

I tried using it with reqwest but reqwest doesn't accept () as a body so try_from fails. I also tried ureq but they want a http::builder instead of the build request.

It would be nice to have an example of how to use sparse-http or maybe even a function that handles the request for you.

note: I know that this doesn't really fit into an issue but I didn't know where to put it else.

[Logarithmus]

Have you tried https://docs.rs/reqwest/latest/reqwest/struct.Request.html#impl-TryFrom%3CRequest%3CT%3E%3E-for-Request?

[ToBinio]

The problem is that reqwest doesn't like () as a body.

I was able to solve that by converting the body to an empty string.

let request = index.make_cache_request("reqwest").unwrap();
let (parts, _) = request.into_parts();
let request = Request::from_parts(parts, "");
let request: reqwest::Request = request.try_into().unwrap();

the next problem is that I couldn't convert the reqwest response back to an HTTP response.

because I don't know the generic type for the HTTP response.

note: crates.io always returned an error (UnsupportedVersion) not sure if this is because of the body conversion.

[ToBinio]

@Jake-Shadle sorry for the ping.

but is there any chance you could help?

[Jake-Shadle]

The problem is that reqwest doesn't like () as a body.

Yes, this is a bug, the implementation in this crate should just use an empty Vec<u8>.

I don't know how you are creating your client, I'd suggest using https://docs.rs/reqwest/0.11.18/reqwest/struct.ClientBuilder.html#method.http2_prior_knowledge since at least crates.io is HTTP/2, and probably any other ones with sparse support like cloudsmith.

I don't use this library any longer, but here is roughly similar code that does the work to convert a request and response using reqwest.

let req = self.index.make_remote_request(name)?;
let req = req.try_into()?;

let res = self.client.execute(req)?;

let mut builder = http::Response::builder()
    .status(res.status())
    .version(res.version());

builder
    .headers_mut()
    .unwrap()
    .extend(res.headers().iter().map(|(k, v)| (k.clone(), v.clone())));

let body = res.bytes()?;
let res = builder.body(body.to_vec())?;

self.index
    .parse_remote_response(name, res, write_cache_entry)

[ToBinio]

@Jake-Shadle Thanks a lot. That worked ❤️

http2_prior_knowledge() sadly did not work for me because it always returned me an frame with invalid size error.

for everyone who needs it here is my complete working code:

use crates_index::SparseIndex;
use http::Request;

fn main() -> color_eyre::eyre::Result<()> {
    let crate_name = "condure";

    let index = SparseIndex::new_cargo_default()?;

    let req = index.make_cache_request(crate_name)?;

    let (parts, _) = req.into_parts();
    let req = Request::from_parts(parts, vec![]);

    let req: reqwest::blocking::Request = req.try_into().unwrap();

    let client = reqwest::blocking::ClientBuilder::new().gzip(true).build()?;

    let res = client.execute(req)?;

    let mut builder = http::Response::builder()
        .status(res.status())
        .version(res.version());

    builder
        .headers_mut()
        .unwrap()
        .extend(res.headers().iter().map(|(k, v)| (k.clone(), v.clone())));

    let body = res.bytes()?;
    let res = builder.body(body.to_vec())?;

    index.parse_cache_response(crate_name, res, true)?;

    Ok(())
}

[dependencies]
color-eyre = "0.6.2"
crates-index = { version = "0.19.13", features = ["sparse-http"] }
http = "0.2.9"
reqwest = { version = "0.11.18", features = ["blocking", "gzip"] }

note: this is just a proof of concept code so it is not very clean

[ToBinio]

This should be resolved with the merge of #131 and the release of 1.0.0