search

storj / edge

Storj edge services (including multi-tenant, S3-compatible server to interact with the Storj network)
GNU Affero General Public License v3.0
48 stars 18 forks source link

Close idle connections gracefully in GMT #354

Open pwilloughby opened 11 months ago

pwilloughby commented 11 months ago

Background

What is the problem/pain point?

Idle connections are currently being terminated by our load balancer with a hard disconnect. FileZilla PRO doesn't handle the hard disconnect well but works fine if the connection is terminated gracefully. 

Status: Resolving address of data.gateway.storjshare.io
Error: GnuTLS error -110 in gnutls_record_recv: The TLS connection was non-properly terminated.
Status: Server did not properly shut down TLS connectionError: Could not read from socket: ECONNABORTED - Connection aborted

There are also some traces available here

Requirements

Close idle connections gracefully. One way to do this would be to have gateway-mt close idle connections before the load balancer does.

Acceptance Criteria

FileZilla Pro no longer gets errors on idle timeout.

pwilloughby commented 10 months ago

The idle timeouts on the load balancer were increased here https://github.com/storj/infra/pull/4738 for an unrelated issue.

nergdron commented 9 months ago

if we make the load balancer have a longer idle timeout than gateway, then would the graceful termination behaviour be enforced, or is there an edge condition I'm not thinking of?

halkyon commented 9 months ago

we did increase the timeouts on haproxy with https://github.com/storj/infra/pull/4738, but then it got reverted due to that authservice client bug where it wouldn't re-use connections, but that's fixed now since v1.61.0. I think we can increase the timeouts again now! @nergdron @pwilloughby

pwilloughby commented 9 months ago

then would the graceful termination behaviour be enforced,

I'm not sure gateway has an idle timeout currently, that probably needs a code or config change.