Open Stono opened 1 month ago
This seems to be a bug in Envoy.
curl -X POST localhost:15000/'drain_listeners?inboundonly'
: works
curl -X POST localhost:15000/'drain_listeners?inboundonly&graceful'
: does not (that is, it applies to outbound traffic)
As a mitigation, i've removed the shutdown hook from our injection template. We already have some reasonable guards to aid with graceful shutdowns that have historically worked ok for us.
The logic in envoy:
if (graceful) {
// Ignore calls to /drain_listeners?graceful if the drain sequence has
// already started.
if (!server_.drainManager().draining()) {
server_.drainManager().startDrainSequence([this, stop_listeners_type, skip_exit]() {
if (!skip_exit) {
server_.listenerManager().stopListeners(stop_listeners_type, {});
}
});
}
} else {
server_.listenerManager().stopListeners(stop_listeners_type, {});
}
You can see in graceful+skip_exit+inboundonly, the stop_listeners_type=inboundonly
is not used anywhere
Is n't this a problem for regular sidecars as well?
No, the prestop hook to drain is only added when native sidecars is enabled
But the same logic gets executed when proxy gets SIGTERM
If that is true then yes, it is a problem there too. The reason I might not have seen it historically is for years we've had a pilot-agent
wrapper script that blocked SIGTERM getting to the proxy and did a while
loop looking for netstat
listening ports from the main app. We removed that "hacky" script as part of the move to native sidecar containers.
Is n't this a problem for regular sidecars as well?
@ramaraochavali I think it is, right? Let's say terminationDrainDuration
is 5min, and drainDuration
is 45s. Presumably, a workload could send a new outbound request after one minute, and the outbound listener would already be shutdown...
yes
Is this the right place to submit this?
Bug Description
When using native sidecars, the following lifecycle is added:
The
preStop
event fires when thePod
enters a terminating state. At this point, the proxy starts sendingconnection: close
for outbound requests, eg the app will still be making outbound calls at this point, but getting aconnection: close
back.You can see this by doing the following:
preStop: sleep 10
to yourapp
containerwatch -n 1 curl <whatever>
kubectl delete the pod
connection: close
headers being returnedThis causes some of our http pool implementations to quite literally implode! :D
So far, I've only got the issue to manifest when the remote destination does not have a sidecar too, for example:
app1 | app1-proxy ----http2-----> remote app (no sidecar)
I do not believe we should be seeing
connection: close
for these outbound requests during the shutdown lifecycle of a pod.Version
Additional Information
No response