reverse proxy instability

[felipenoris]

Hi! As I mentioned in https://github.com/actix/actix-web/issues/553, I´m working on a derivative of this repo at https://github.com/felipenoris/actix-reverse-proxy .

I´m facing a few difficulties and would like to ask for some thoughts.

First, a few improvements comparing to this repo:

• I´m creating x-forwarded-for header
• I´m setting .no_default_headers() in the forward_req.

Without these changes the reverse proxy doesn´t even work for me. So maybe you could benefit from these improvements.

I also tried to implement correct handling of hop-by-hop headers, as in https://github.com/brendanzab/hyper-reverse-proxy and https://golang.org/src/net/http/httputil/reverseproxy.go

My setup is: nginx <-> reverse_proxy_1 <-> reverse_proxy_2 <-> HTTP service (both reverse proxies are Rust actix-web based).

The problems I´m getting are:

• the HTTP service is getting the header transfer-encoding=chunked. When I debug the forward_req this header doesn´t show up. Why do this get created? This is a hop-by-hop header, so I guess I should remove this header.

• sometimes I can´t get a correct response from HTTP service. The browser just reports HTTP ERROR 500 and reverse_proxy_2 reports an error: Error: IO error: Connection reset by peer (os error 104).

Any thoughts on how to fix these problems?

[DoumanAsh]

Oh I see. Just to clarify I mostly made this repo to illustrate how to forward client requests by using server requests.

But your improvements are certainly required for proper proxy.

the HTTP service is getting the header transfer-encoding=chunked. When I debug the forward_req this header doesn´t show up. Why do this get created? This is a hop-by-hop header, so I guess I should remove this header.

Can you check if this header present in server request? You can copy-paste my logger middleware to get a better look at headers

sometimes I can´t get a correct response from HTTP service

reverse_proxy_2 is Rust's proxy, right? Do you see that it gets response from proxy_1 before error or?

[felipenoris]

Hey, thanks for you help!

Both reverse proxies are Rust ´s proxy using actix-reverse-proxy.

I worked out the debugging and this is what I got, in the order of the events:

[ browser ] --> SEND: request to NGINX

[ NGINX ] --> RECEIVE browser request --> SEND request to REV-PROXY-1

[ REV-PROXY-1] --> RECEIVE request without transfer-encoding: chunked header --> SEND request to REV-PROXY-2 without transfer-encoding: chunked header

[ REV-PROXY-2] --> RECEIVE request with transfer-encoding: chunked header --> Removes transfer-encoding header and SEND to SERVICE

[ SERVICE] --> RECEIVE request with transfer-encoding: chunked header --> SEND response to REV-PROXY-2

[ REV-PROXY-2] --> RECEIVE response with headers: content-length, content-type, date --> SEND response to REV-PROXY-1

[REV-PROXY-1] --> RECEIVE response with headers: content-encoding=br, content-type, date. --> SEND response to NGINX

[NGINX] --> RECEIVE response --> SEND response to Browser

[Browser] --> Receive response with headers:

Connection: keep-alive
content-encoding: br
Content-Type: text/plain; charset=utf-8
Date: Wed, 24 Oct 2018 14:39:25 GMT
Server: nginx
Transfer-Encoding: chunked

[felipenoris]

Debug analysis

Although the debugger from REV-PROXY-1 doesn´t say that it generated header transfer-encoding, REV-PROXY-2 gets this header. So I guess this gets generated when the forward request is sent.

Is this because the body is set to streaming mode?

let mut forward_req = forward_req
                                    .no_default_headers()
                                    .set_header_if_none(actix_web::http::header::USER_AGENT, "")
                                    .body(actix_web::Body::Streaming(Box::new(forward_body)))
                                    .expect("To create valid forward request");

Back response is also set to streaming mode:

let mut back_rsp = back_rsp
                            .body(actix_web::Body::Streaming(Box::new(back_body)));

I tried to compare this to Go´s implementation, but it´s not clear to me that it does streaming. I guess it uses, by default, the RoundTripper Transport.

res, err := transport.RoundTrip(outreq)
    if err != nil {
        p.getErrorHandler()(rw, outreq, err)
        return
    }

Should I try to disable streaming? How can I do that in actix?

Error case

So, in the error scenario, what happens is that REV-PROXY-1 reports Error: IO error: Connection reset by peer (os error 104) and REV-PROXY-2 doesn´t even get the request.

[DoumanAsh]

Small clarification does proxy 2 uses HTTP2? Afaik we do not set transfer-encoding for HTTP2 due to h2 library

[felipenoris]

I do not set any version for HTTP. Looking at the logs, it appears HTTP/1.1 in the requests on every node.

[DoumanAsh]

As far as I remember in case of HTTP1 it should set chunked encoding, but out of curiosity did you try to set it manually?

[felipenoris]

You mean to set HTTP1 or chunked?

[DoumanAsh]

chunked

But you may also try to disable HTTP2 if you use SSL

[DoumanAsh]

Btw, following your advice I updated headers handling https://github.com/DoumanAsh/actix-reverse-proxy/commit/65cbfb9452ce711cccda5c89dfe20b625e34db81

Though now I think to complete it I'd need to handle forwarded header too

[felipenoris]

I set .chunked() in the client requests and the headers in each node are the same (including transfer-encoding), but the round-trip takes 5 seconds to complete (it was supposed to take less than 100 miliseconds). Everything gets very unstable.

Then I set .no_http2 in the rust proxy servers, and everything was the same...

[DoumanAsh]

Do you see where delay happens?

Also, if you use gitter, you can PM me there, we could just chat there

[felipenoris]

I'll post here the updates on this, in case someone else wants to help.

So, based on this setup: (both rev-proxy-1 and rev-proxy-2 are Rust actix-web based)

[ browser ] <-> [ NGINX ] <-> [ rev-proxy-1] <-> [rev-proxy-2 ] <-> [service]

If I set .chunked() in the client requests, the delay I get is in the response from [rev-proxy-2] to [rev-proxy-1]. In this case, [rev-proxy-2] responds quickly, but [rev-proxy-1] takes some time to read it.

I also noticed that this only happens when I'm accessing from an outside network. When using inside network, everything works the same as before, but with intermitent IO errors reported by [rev-proxy-1].

The only difference I see between inside/outside networks is the size of the request: from inside network the request is bigger because it includes user identification inside cookies.

I also tried disabling the removal of hop-by-hop headers. If I do this, I always get ERR_INCOMPLETE_CHUNKED_ENCODING when accessing from outside network. Everything works the same from inside network, with intermitent IO errors.

I also tracked the places where Content-Lenght header appears. It only appears in the response from [service] to [rev-proxy-2]. It does not get sent back to [rev-proxy-1] , and all the way to the browser.

I have the same setup implemented using Go's default reverse proxy. Everything works fine, and the only difference I get is that there are no transfer-encoding headers anywhere, as far as I can tell.

All this makes me think that the problem might be related to chunking and the Content-Lenght headers. All these get generated during the streaming and I don't know how to change this behavior.

[felipenoris]

cc @fafhrd91.