pingcap / tiproxy

Apache License 2.0
56 stars 27 forks source link

go client reports `invalid connection` after idle for wait_timeout #327

Open djshow832 opened 1 year ago

djshow832 commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

dbt.mustExec("SET @@SESSION.wait_timeout = 2")

// wait for TiDB to close our connection
time.Sleep(3 * time.Second)

tx, err := dbt.db.Begin()
if err != nil {
    dbt.Fatal(err)
}

2. What did you expect to see? (Required)

The driver logs packets.go:122: closing bad idle connection: EOF 3 times but dbt.db.Begin doesn't report any errors because it creates a new connection internally and then retry.

3. What did you see instead (Required)

The driver logs packets.go:37: unexpected EOF and dbt.db.Begin reports invalid connection.

4. What is your version? (Required)

master.

The reason: TiProxy doesn't close the connection immediately after wait_timeout and still receives requests. After it receives the request, it finds that TiDB has disconnected and then closes the client connection. So the client encounters an ErrInvalidConn in readPacket instead of ErrBadConn in writePacket. The client will retry by creating a new connection if encounters ErrBadConn but won't retry if encounters ErrInvalidConn, see https://github.com/golang/go/blob/master/src/database/sql/sql.go#L1536-1546 I think there's no way to solve it.

djshow832 commented 1 year ago

Maybe a workaround is to add a client idle timeout to tiproxy configuration, like client timeout in haproxy: https://www.papertrail.com/solution/tips/haproxy-logging-how-to-tune-timeouts-for-performance/