search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

Http/1.0 client attempting keep-alive gets back "Connection: close" #1651

Closed tmack8001 closed 7 years ago

tmack8001 commented 7 years ago

In testing things today I noticed that when a client is using http/1.0 and attempting to use keep-alive Zuul (setup with spring-cloud-netflix) returns a "Connection: closed" header. I'm have a filter similar to the NFPostDecoration filter (https://github.com/Netflix/zuul/blob/939ada9dce57709cf6b250b77fabfa37e8d988bf/zuul-netflix-webapp/src/main/groovy/filters/post/PostDecoration.groovy#L61) that is attempting to force keep-alive from the zuul server side. The forcing of keep-alive works for http/1.1 clients, but has a bug in http/1.0 where there are two "Connection" headers (close, and keep-alive) as well

Here is a way to reproduce this using curl, found the issues when attempting a benchmark using AB (ApacheBench/2.3).

$ curl -v --http1.0 -H "Connection: keep-alive"  "http://localhost:8080/nginx"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /nginx HTTP/1.0
> Host: localhost:8080
> User-Agent: curl/7.51.0
> Accept: */*
> Connection: keep-alive
>
< HTTP/1.1 200
< X-Application-Context: Stargate Zuul:8080
< X-Zuul-Debug-Header: [[[Filter pre 5 PreDecorationFilter]]][[[Filter {PreDecorationFilter TYPE:pre ORDER:5} Execution time = 2ms]]][[[{PreDecorationFilter} added ignoredHeaders=[authorization, set-cookie, cookie]]]][[[{PreDecorationFilter} added originResponseHeaders=[com.netflix.util.Pair@5bef2aa5]]]][[[{PreDecorationFilter} added zuulRequestHeaders={x-forwarded-host=localhost:8080, x-forwarded-proto=http, x-forwarded-prefix=/nginx, x-forwarded-port=8080, x-forwarded-for=172.25.0.1}]]][[[{PreDecorationFilter} added requestURI=]]][[[{PreDecorationFilter} added proxy=nginx]]][[[{PreDecorationFilter} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][1ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms]]]][[[{PreDecorationFilter} added serviceId=nginx]]][[[Filter pre 20 PreDecoration]]][[[Filter {PreDecoration TYPE:pre ORDER:20} Execution time = 1ms]]][[[{PreDecoration} changed zuulRequestHeaders={x-request-id=6077ce8b-3aeb-434b-8f85-9101cdb04dc5, x-forwarded-host=localhost:8080, x-forwarded-proto=http, x-forwarded-prefix=/nginx, x-forwarded-port=8080, x-forwarded-for=172.25.0.1}]]][[[{PreDecoration} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][1ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][1ms]]]][[[Invoking {route} type filters]]][[[Filter route 10 RibbonRoutingFilter]]][[[Filter {RibbonRoutingFilter TYPE:route ORDER:10} Execution time = 5ms]]][[[{RibbonRoutingFilter} changed originResponseHeaders=[com.netflix.util.Pair@5bef2aa5, com.netflix.util.Pair@354ba130, com.netflix.util.Pair@8e075f30, com.netflix.util.Pair@b79f65e8, com.netflix.util.Pair@5b193a75, com.netflix.util.Pair@ec79757b, com.netflix.util.Pair@8a51e22e, com.netflix.util.Pair@67245cc0, com.netflix.util.Pair@c88a6d0]]]][[[{RibbonRoutingFilter} added responseDataStream=org.apache.http.conn.EofSensorInputStream@42b76cd8]]][[[{RibbonRoutingFilter} added zuulResponseHeaders=[com.netflix.util.Pair@354ba130, com.netflix.util.Pair@5b193a75, com.netflix.util.Pair@ec79757b, com.netflix.util.Pair@67245cc0, com.netflix.util.Pair@c88a6d0]]]][[[{RibbonRoutingFilter} added responseStatusCode=200]]][[[{RibbonRoutingFilter} added chunkedRequestBody=true]]][[[{RibbonRoutingFilter} added responseGZipped=false]]][[[{RibbonRoutingFilter} added ribbonResponse=org.springframework.cloud.netflix.ribbon.apache.RibbonApacheHttpResponse@6ab526ef]]][[[{RibbonRoutingFilter} added originContentLength=612]]][[[{RibbonRoutingFilter} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][1ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][1ms], RibbonRoutingFilter[SUCCESS][5ms]]]][[[Filter route 100 SimpleHostRoutingFilter]]][[[Filter route 500 SendForwardFilter]]][[[Invoking {post} type filters]]][[[Filter post -2147483648 PostDecoration]]][[[Filter {PostDecoration TYPE:post ORDER:-2147483648} Execution time = 97699ms]]][[[{PostDecoration} changed zuulResponseHeaders=[com.netflix.util.Pair@354ba130, com.netflix.util.Pair@5b193a75, com.netflix.util.Pair@ec79757b, com.netflix.util.Pair@67245cc0, com.netflix.util.Pair@c88a6d0, com.netflix.util.Pair@6a2361, com.netflix.util.Pair@ea43898f, com.netflix.util.Pair@839fc313, com.netflix.util.Pair@be8cc939, com.netflix.util.Pair@542ba202, com.netflix.util.Pair@9d224c56, com.netflix.util.Pair@c400bbcb, com.netflix.util.Pair@8066b23d, com.netflix.util.Pair@890cfc55, com.netflix.util.Pair@8e4ff552, com.netflix.util.Pair@b7a8c594, com.netflix.util.Pair@31d61e92, com.netflix.util.Pair@8a51e22e, com.netflix.util.Pair@1f196e61]]]][[[{PostDecoration} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][1ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][1ms], RibbonRoutingFilter[SUCCESS][5ms], PostDecoration[SUCCESS][97699ms]]]][[[Filter post 0 SendErrorFilter]]][[[Filter post 1000 SendResponseFilter]]]
< Date: Thu, 26 Jan 2017 06:56:35 GMT
< Last-Modified: Tue, 27 Dec 2016 14:23:08 GMT
< ETag: "5862794c-264"
< Accept-Ranges: bytes
< Via: zuul
< X-Zuul: zuul
< X-Zuul-instance: unknown
< Origin-X-Zuul-ServiceId: nginx
< Origin-Date: Thu, 26 Jan 2017 06:56:35 GMT
< Origin-Server: nginx/1.11.8
< Origin-Content-Length: 612
< Origin-Last-Modified: Tue, 27 Dec 2016 14:23:08 GMT
< Origin-Content-Type: text/html
< Origin-Connection: keep-alive
< Origin-ETag: "5862794c-264"
< Origin-Accept-Ranges: bytes
< Connection: keep-alive
< X-Zuul-Filter-Executions: ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][1ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][1ms], RibbonRoutingFilter[SUCCESS][5ms]
< Content-Type: text/html
< Connection: close
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Curl_http_done: called premature == 0
* Closing connection 0

Here is how I have modified my Filter to "force" AB to be happy with the response and it also seems to allow keep-alive on Http/1.0 which requires a Content-Length header and further more Http/1.0 doesn't support chunked encoding.

// is this correct behavior?
        if (KEEP_ALIVE.equalsIgnoreCase(request.getHeader(CONNECTION))) {
            // hack to set content-length when using http/1.0 keep-alive
            if (request.getProtocol().equalsIgnoreCase("http/1.0") && context.isChunkedRequestBody()) {
                context.set("chunkedRequestBody", Boolean.FALSE);
                headers.add(new Pair<>("Content-Length", "" + 612));
            } else {
                // seems that the above code will set keep-alive already
                headers.add(new Pair<>(CONNECTION, KEEP_ALIVE));
            }
        } else {
            // seems like the above already sets keep-alive
            headers.add(new Pair<>(CONNECTION, KEEP_ALIVE));
        }

After adding this caused AB did a benchmark successfully making 99% keep-alive requests and also makes it so that there is only one Connection header being returned by Zuul (the keep alive one I'm attempting to force).

$ ab -c 10 -t 60 -v 1 -k localhost:8080/nginx
This is ApacheBench, Version 2.3 <$Revision: 1748469 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Finished 12567 requests

Server Software:        nginx/1.11.8
Server Hostname:        localhost
Server Port:            8080

Document Path:          /nginx
Document Length:        612 bytes

Concurrency Level:      10
Time taken for tests:   60.002 seconds
Complete requests:      12567
Failed requests:        117
   (Connect: 0, Receive: 0, Length: 117, Exceptions: 0)
Keep-Alive requests:    12447
Total transferred:      72526373 bytes
HTML transferred:       7619400 bytes
Requests per second:    209.44 [#/sec] (mean)
Time per request:       47.746 [ms] (mean)
Time per request:       4.775 [ms] (mean, across all concurrent requests)
Transfer rate:          1180.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0   48  16.4     46     171
Waiting:        0   48  16.5     46     171
Total:          0   48  16.3     46     171

Percentage of the requests served within a certain time (ms)
  50%     46
  66%     53
  75%     57
  80%     60
  90%     68
  95%     77
  98%     87
  99%     96
 100%    171 (longest request)
$ curl -v --http1.0 -H "Connection: keep-alive"  "http://localhost:8080/nginx"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /nginx HTTP/1.0
> Host: localhost:8080
> User-Agent: curl/7.51.0
> Accept: */*
> Connection: keep-alive
>
< HTTP/1.1 200
< X-Application-Context: Stargate Zuul:8080
< X-Zuul-Debug-Header: [[[Filter pre 5 PreDecorationFilter]]][[[Filter {PreDecorationFilter TYPE:pre ORDER:5} Execution time = 2ms]]][[[{PreDecorationFilter} added ignoredHeaders=[authorization, set-cookie, cookie]]]][[[{PreDecorationFilter} added originResponseHeaders=[com.netflix.util.Pair@5bef2aa5]]]][[[{PreDecorationFilter} added zuulRequestHeaders={x-forwarded-host=localhost:8080, x-forwarded-proto=http, x-forwarded-prefix=/nginx, x-forwarded-port=8080, x-forwarded-for=172.25.0.1}]]][[[{PreDecorationFilter} added requestURI=]]][[[{PreDecorationFilter} added proxy=nginx]]][[[{PreDecorationFilter} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][0ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms]]]][[[{PreDecorationFilter} added serviceId=nginx]]][[[Filter pre 20 PreDecoration]]][[[Filter {PreDecoration TYPE:pre ORDER:20} Execution time = 0ms]]][[[{PreDecoration} changed zuulRequestHeaders={x-request-id=b398b5cc-38b5-4639-b990-5ac20c718562, x-forwarded-host=localhost:8080, x-forwarded-proto=http, x-forwarded-prefix=/nginx, x-forwarded-port=8080, x-forwarded-for=172.25.0.1}]]][[[{PreDecoration} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][0ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][0ms]]]][[[Invoking {route} type filters]]][[[Filter route 10 RibbonRoutingFilter]]][[[Filter {RibbonRoutingFilter TYPE:route ORDER:10} Execution time = 4ms]]][[[{RibbonRoutingFilter} changed originResponseHeaders=[com.netflix.util.Pair@5bef2aa5, com.netflix.util.Pair@76b89ad, com.netflix.util.Pair@8e075f30, com.netflix.util.Pair@b79f65e8, com.netflix.util.Pair@5b193a75, com.netflix.util.Pair@ec79757b, com.netflix.util.Pair@8a51e22e, com.netflix.util.Pair@67245cc0, com.netflix.util.Pair@c88a6d0]]]][[[{RibbonRoutingFilter} added responseDataStream=org.apache.http.conn.EofSensorInputStream@603330d4]]][[[{RibbonRoutingFilter} added zuulResponseHeaders=[com.netflix.util.Pair@76b89ad, com.netflix.util.Pair@5b193a75, com.netflix.util.Pair@ec79757b, com.netflix.util.Pair@67245cc0, com.netflix.util.Pair@c88a6d0]]]][[[{RibbonRoutingFilter} added responseStatusCode=200]]][[[{RibbonRoutingFilter} added chunkedRequestBody=true]]][[[{RibbonRoutingFilter} added responseGZipped=false]]][[[{RibbonRoutingFilter} added ribbonResponse=org.springframework.cloud.netflix.ribbon.apache.RibbonApacheHttpResponse@119af6f8]]][[[{RibbonRoutingFilter} added originContentLength=612]]][[[{RibbonRoutingFilter} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][0ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][0ms], RibbonRoutingFilter[SUCCESS][4ms]]]][[[Filter route 100 SimpleHostRoutingFilter]]][[[Filter route 500 SendForwardFilter]]][[[Invoking {post} type filters]]][[[Filter post -2147483648 PostDecoration]]][[[Filter {PostDecoration TYPE:post ORDER:-2147483648} Execution time = 1ms]]][[[{PostDecoration} changed zuulResponseHeaders=[com.netflix.util.Pair@76b89ad, com.netflix.util.Pair@5b193a75, com.netflix.util.Pair@ec79757b, com.netflix.util.Pair@67245cc0, com.netflix.util.Pair@c88a6d0, com.netflix.util.Pair@6a2361, com.netflix.util.Pair@ea43898f, com.netflix.util.Pair@839fc313, com.netflix.util.Pair@be8cc939, com.netflix.util.Pair@264b8a8f, com.netflix.util.Pair@9d224c56, com.netflix.util.Pair@c400bbcb, com.netflix.util.Pair@8066b23d, com.netflix.util.Pair@890cfc55, com.netflix.util.Pair@8e4ff552, com.netflix.util.Pair@b7a8c594, com.netflix.util.Pair@31d61e92, com.netflix.util.Pair@b79f65e8, com.netflix.util.Pair@c1f8f2a1]]]][[[{PostDecoration} changed chunkedRequestBody=false]]][[[{PostDecoration} changed executedFilters=ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][0ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][0ms], RibbonRoutingFilter[SUCCESS][4ms], PostDecoration[SUCCESS][1ms]]]][[[Filter post 0 SendErrorFilter]]][[[Filter post 1000 SendResponseFilter]]]
< Date: Thu, 26 Jan 2017 07:21:40 GMT
< Last-Modified: Tue, 27 Dec 2016 14:23:08 GMT
< ETag: "5862794c-264"
< Accept-Ranges: bytes
< Via: zuul
< X-Zuul: zuul
< X-Zuul-instance: unknown
< Origin-X-Zuul-ServiceId: nginx
< Origin-Date: Thu, 26 Jan 2017 07:21:40 GMT
< Origin-Server: nginx/1.11.8
< Origin-Content-Length: 612
< Origin-Last-Modified: Tue, 27 Dec 2016 14:23:08 GMT
< Origin-Content-Type: text/html
< Origin-Connection: keep-alive
< Origin-ETag: "5862794c-264"
< Origin-Accept-Ranges: bytes
< X-Zuul-Filter-Executions: ServletDetectionFilter[SUCCESS][0ms], Servlet30WrapperFilter[SUCCESS][0ms], RateLimitFilter[SUCCESS][0ms], PreFilter[SUCCESS][0ms], DebugFilter[SUCCESS][0ms], PreDecorationFilter[SUCCESS][2ms], PreDecoration[SUCCESS][0ms], RibbonRoutingFilter[SUCCESS][4ms]
< Content-Type: text/html
< Content-Length: 612
< Connection: keep-alive
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
tmack8001 commented 7 years ago

Obviously headers.add(new Pair<>("Content-Length", "" + 612)); isn't what I want to have here, this was just a quick hack to see if things were going to "work" if that header was present... and it did do the job successfully.

UPDATE: to get the actual response size added in the header I changed the above to headers.add(new Pair<>("Content-Length", "" + context.getOriginContentLength()));

ryanjbaxter commented 7 years ago

Is this a problem with Spring Cloud or Zuul? From your description it seems the problem is with Zuul and not anything in the Spring Cloud library.

tmack8001 commented 7 years ago

At the moment I'm not actually sure. In my debugging yesterday wasn't able to find where the "Connection: Closed" was coming from nor why a HTTP/1.0 request was being marked for chunked encoding.

tmack8001 commented 7 years ago

You are right after launching the zuul (vanilla) with eureka and a sample hello app I still see this type of thing. So will open a ticket over with netflix/zuul