aws / aws-xray-daemon

The AWS X-Ray daemon listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray API.
Apache License 2.0
191 stars 69 forks source link

Add http2 timouts to close bad TCP connection #216

Closed wangzlei closed 9 months ago

wangzlei commented 9 months ago

Issue #, if available:

During ALB maintainance, there could be cases where the TCP connection from Daemon to X-Ray ALB is not properly closed, the Daemon client will not know the TCP connection is no longer working, and will be kept using it for sending new HTTP2 requests as HTTP2 multplexes all the requests in a single TCP connection, this would results in repeated request timeouts and evetually losing data.

Description of changes:

In this change 2 timeouts has been added:

So in the case of a TCP connection not being closed properly, the http2 transport would detect there is no frame is being received, then send out a Ping, and if no response is received after the PintTimeout, it would close the connection. Existing pending requests would receive an connection error, which would be retried, and subsequent requests would result in a new connection being created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.