How to modify response body?

zu1k commented 3 years ago

Could you please tell me how to modify the response?

I tried to get the specific content of the body in the closure and convert it to a string, but now I can't await it in the closure.

omjadas commented 3 years ago

Hi @zu1k,

Thanks for reaching out. I have created a branch that has changed the return types for the request and response handlers so that async blocks can be more easily used inside of them. I will most likely merge it to main and do a release in a couple days.

Once merged something like the following should be able to satisfy this need.

let response_handler = |res: Response<Body>| -> Pin<Box<dyn Future<Output = Response<Body>> + Send>> {
    Box::pin(async {
        let (parts, body) = res.into_parts();
        let body_bytes = hyper::body::to_bytes(body).await.unwrap();
        let new_bytes = body_bytes; // Do something with body_bytes
        let res = Response::from_parts(parts, Body::from(new_bytes));
        res
    })
};

zu1k commented 3 years ago

I noticed that hyper is a pretty low-level lib, if the content encoding is gzip, we can't modified it easily.

Is there any way to deal with this case? What if we use high level libraries like wrap and reqwest?

omjadas commented 3 years ago

I think the best way to deal with encoded bodies is to leave it to the user to use a library such as flate2 or brotli.

An alternative would be to delete the accept-encoding header from all requests in the request handler.

zu1k commented 3 years ago

An alternative would be to delete the accept-encoding header from all requests in the request handler.

I will try this, thank you.

omjadas commented 3 years ago

I have released this change as part of v0.3.0.

zu1k commented 3 years ago

I think the best way to deal with encoded bodies is to leave it to the user to use a library such as flate2 or brotli.

I haved tried to use these crates to deal with encoded bodies, I found it a little complicated, most of the code is duplicated with the reqwest crate.

An alternative would be to delete the accept-encoding header from all requests in the request handler.

I also tried this method, most sites work fine, a small number of websites do not comply with the specifications and still return encoded bodies, and the returned headers still indicate the use of gzip.

Now I'm trying to use my super client(reqwest) to do request, and return response directly in the process of hooking request. It works good, however, it is still necessary to write a lot of compatible code to convert hyper req to reqwest req as well as to convert resp.

I think it is more appropriate to transfer this part of the code to the crate. The users of the crate should only focus on the tasks they want to achieve.

So I still strongly recommend to use reqwest as the client in your crate, and use reqwest's req or resp in the parameters passed to the closure, which provides more convenient methods.

Waiting for you to further consider this matter, thank you. @omjadas

omjadas commented 3 years ago

I'm not sure if using reqwest is the best option, as you have mentioned the requests and responses would have to be converted between the different types. I would also be concerned about the performance impact of reqwest decoding every response, even if the user does not care about the body for a particular response. I am thinking that perhaps a function can be provided that the user can call in their response handler that will take care of decoding the body, something like the following.

pub fn decode_response(res: Response<Body>) -> Result<Response<Body>, Error> {
    panic!("not implemented");
}

#[async_trait]
impl HttpHandler for LogHandler {
    async fn handle_request(&mut self, _ctx: &HttpContext, req: Request<Body>) -> RequestOrResponse {
        RequestOrResponse::Request(req)
    }

    async fn handle_response(&mut self, _ctx: &HttpContext, res: Response<Body>) -> Response<Body> {
        let res = decode_response(res).unwrap();
        // do something with decoded response
        res
    }
}

omjadas commented 3 years ago

@zu1k, I have released v0.4.1, which includes the above-mentioned function. Let me know whether this helps with your use cases.

zu1k commented 3 years ago

Thank you very much for your awesome crate, this provides strong power for the application that I'm developing recently.

I'm a novice to Rust, every time you add features, it brings me a suprise. I learned a lot from your crate, thank you very much for both your work and your very prompt reply.

zu1k commented 3 years ago

Let me know whether this helps with your use cases.

I just tested it and it works, thank you.

https://github.com/zu1k/good-mitm

omjadas / hudsucker

How to modify response body? #5