search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.68k stars 4.75k forks source link

Graceful shutdown with HTTP2 CONNECT #34897

Open howardjohn opened 2 months ago

howardjohn commented 2 months ago

Title: Graceful shutdown with HTTP2 CONNECT

Description: We use Envoy to tunnel HTTP traffic over HTTP2 CONNECT.

When the server is shutting down, it will send us a GOAWAY. However, Envoy will still treat the connection as live.

What we want is for the inner-HTTP to stop using the HTTP2 CONNECT transport when it gets a GOAWAY, and only send new inner-http requests on a new CONNECT tunnel.

mattklein123 commented 2 months ago

cc @alyssawilk

alyssawilk commented 2 months ago

Is this one of those "two passes through Envoy" issues?

The HTTP connection pool does properly handle goaways by marking the connection as draining and not creating new streams on that connection https://github.com/envoyproxy/envoy/blob/main/source/common/http/conn_pool_base.cc#L100

That said that HTTP/2 connection has no idea it's handling HTTP/1.1 as the inner payload so can't send a connection: close header. I think that'd have to be done by whoever is sending the HTTP/1.1. If you want to handle it in Envoy you could potentially advise open streams that a connection is draining and propagate that through the internal listener code but that's a feature I suspect you'd have to pick up yourselves.

howardjohn commented 2 months ago

I don't think I necessarily need connection:close onto the HTTP 1/1 connections.

So we have 

app -- HTTP/1.1 ---> Envoy (Handling HTTP with an HCM) ---Internal Listener--> Same Envoy process ---> HTTP/2 CONNECT ---> Backend Proxy server (Terminates CONNECT) ---HTTP/1.1 --> backend app

When the "Backend Proxy server" is shutting down, it will send a GOAWAY over the outer HTTP/2 CONNECT, wait N seconds, then close the connection.

During this time between the GOAWAY and connection close ("draining"), I would want Envoy to not use the draining CONNECT connection for new HTTP/1.1 requests, and instead open up a new CONNECT request (or use an existing one, if there is another one already).

alyssawilk commented 2 months ago

yeah so you'd want to tweak the codec client to warn all streams when the connection is draining, propogate that through the internal listener, and use that as a drain signal for the HTTP/1 "connection"