Intermittent 426 Upgrade required

[nabeelio]

Hi all, I'm having a strange issue. I'm running Ambassador with Envoy, and there are intermittent 426 errors returned. I'm not quite sure what to make of it. I'm using an Azure internal load balancer, which is on a static IP.

The endpoint is /api/v1/gateway, which just returns a 200, and a JSON string. Nothing fancy.

I did see that @huxiaobabaer had a similar issue, but it was closed. Here's some requests:

ACCESS [2018-11-14T18:17:06.296Z] "GET /api/v1/gateway HTTP/1.0" 426 - 0 0 0 - "-" "-" "-" "10.4.0.30" "-"

However, tcpdump shows:

18:17:06.295890 IP 10.240.0.4.24882 > ambassador-67dcc99954-g9fgg.80: Flags [P.], seq 6916:7007, ack 7905, win 1024, length 91: HTTP: GET /api/v1/gateway HTTP/1.1
18:17:06.296270 IP ambassador-67dcc99954-g9fgg.80 > 10.240.0.4.24882: Flags [P.], seq 7905:8009, ack 7007, win 237, length 104: HTTP: HTTP/1.1 426 Upgrade Required

And I've started getting this in my browser as well:

18:58:17.043326 IP (tos 0x0, ttl 126, id 29750, offset 0, flags [DF], proto TCP (6), length 499)
    10.240.0.4.25999 > ambassador-67dcc99954-g9fgg.80: Flags [P.], cksum 0xbd5c (correct), seq 2920:3379, ack 2425, win 1025, length 459: HTTP, length: 459
    OPTIONS / HTTP/1.1
    Connection: Keep-Alive
    Host: 10.240.1.10
    Max-Forwards: 10
    User-Agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
    Origin: example.com
    Access-Control-Request-Method: CONNECT
    X-FORWARDED-PROTO: http
    X-FORWARDED-PORT: 80
    X-ORIGINAL-HOST: 98d7d566-9097-44a3-9db2-bfd229f405bf.cloudapp.net
    X-Original-URL: /
    X-Forwarded-For: [EDITED]
    X-ARR-LOG-ID: a2b7c9d9-ed13-4603-8b4c-db638dcc3ee4

18:58:17.043706 IP (tos 0x0, ttl 64, id 64491, offset 0, flags [DF], proto TCP (6), length 144)
    ambassador-67dcc99954-g9fgg.80 > 10.240.0.4.25999: Flags [P.], cksum 0x16a5 (incorrect -> 0x145d), seq 2425:2529, ack 3379, win 303, length 104: HTTP, length: 104
    HTTP/1.1 426 Upgrade Required
    date: Wed, 14 Nov 2018 18:58:16 GMT
    server: envoy
    content-length: 0

And the corresponding Envoy log:

':authority', '10.240.1.10'
':path', '/'
':method', 'OPTIONS'
'connection', 'Keep-Alive'
'max-forwards', '10'
'user-agent', 'Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)'
'origin', 'example.com'
'access-control-request-method', 'CONNECT'
'x-forwarded-proto', 'http'
'x-forwarded-port', '80'
'x-original-host', '98d7d566-9097-44a3-9db2-bfd229f405bf.cloudapp.net'
'x-original-url', '/'
'x-forwarded-for', '[EDITED]'
'x-arr-log-id', 'a2b7c9d9-ed13-4603-8b4c-db638dcc3ee4'

[2018-11-14 18:58:17.043][67][debug][http] source/common/http/conn_manager_impl.cc:984] [C67648][S2895827541538762879] encoding headers via codec (end_stream=true):
':status', '426'
'date', 'Wed, 14 Nov 2018 18:58:16 GMT'
'server', 'envoy'

The odd part is that when I restart Envoy, it behaves for a little bit. I'm running tcpdump in the same container that Envoy is running in.

Running a kubectl port-forward ... to bypass Envoy works fine. So I'm not sure where this HTTP/1.0 downgrade is coming from and I'm not sure where to go from here. Looking through the Envoy source, I only see one instance of where this response is sent.

I've attached some of the stats/logs. I couldn't run the collector - when I restart it, it's intermittent so I'm not sure how large the logs would get - it's also running in a container

Now, I know this is an older version of Envoy, I can try upgrading to the latest Ambassador EA 5, but I'm not able to do that at the moment.

Thanks for the help!

diag.txt

[daimoniac]

In my case, the HTTP/1.0 requests are health checks issued by digitaloceans loadbalancer. We need to be able to say "OK" to these health checks, or the service will be marked as DOWN.

[oschaaf]

@daimoniac It looks like http/1.0 is supported, but it needs explicit configuration: https://github.com/envoyproxy/envoy/blob/02b2e798593ecebc48ef7c4a4f1832d5e533498d/source/common/http/conn_manager_impl.cc#L568

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.