benweint
commented
1 year ago Ok, here's a test case:
package main
import (
"context"
"fmt"
"github.com/jackc/pgx/v5/pgxpool"
"os"
"sync"
"time"
)
func main() {
// urlExample := "postgres://username:password@localhost:5432/database_name"
ctx := context.Background()
cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
if err != nil {
panic(err)
}
cfg.MinConns = 100
cfg.MaxConns = 100
pool, err := pgxpool.NewWithConfig(context.Background(), cfg)
if err != nil {
panic(err)
}
defer pool.Close()
concurrency := 100
var wg sync.WaitGroup
for i := 0; i < concurrency; i++ {
wg.Add(1)
go runQueries(ctx, fmt.Sprintf("%d", i), pool)
}
wg.Wait()
}
func runQueries(ctx context.Context, id string, pool *pgxpool.Pool) {
for {
runOnce(ctx, id, pool)
}
}
func runOnce(ctx context.Context, id string, pool *pgxpool.Pool) {
ctx, cancel := context.WithTimeout(ctx, 1*time.Second)
defer cancel()
var one int
err := pool.QueryRow(ctx, "SELECT 1").Scan(&one)
if err != nil {
fmt.Printf("got error in %s: %v\n", id, err)
return
}
fmt.Printf("ok %s!\n", id)
}Run with:
DATABASE_URL=postgres://<user>:<pass>@localhost:<port>/<db> go run .If connected directly to a DB, works fine. To reproduce this issue, the easiest thing I've found so far is to use Toxiproxy:
Start the server with toxiproxy-server. Create a proxy listening on some port, pointing to your local Postgres:
toxiproxy-cli create --listen localhost:5435 --upstream localhost:5432 pgRun the test case again, pointed at the proxy listening port (5435 in this example) instead of directly at the DB. Verify that it works (you should see lots of ok! messages).
Once it's been running for a bit, add a timeout toxic like this:
toxiproxy-cli toxic add --type timeout --attribute timeout=3 --toxicName to pgLet the test case continue running, and watch it eventually grind to a halt. You can remove the toxic like this:
toxiproxy-cli toxic remove --toxicName to pg ... but the app is now dead and won't recover from this state.
jackc
jon-whit
jorgerasillo
hochhaus
Describe the bug We're seeing an issue wherein we can consistently reproduce application hangs with
pgx/v5that do not occur withpgx/v4. The only changes to the application itself between the pgx/v4-based version that works and the pgx/v5 version that doesn't are those necessitated by the API differences between the two versions (namely,pgxpool.ConnectConfigvs.pgxpool.NewWithConfig).To Reproduce
I don't have a compact and easily shareable reproduction case yet, but here's the outline of what we're doing:
pgx/v5to read from a Postgres 14 database. In our case, the DB happens to be running on GCP's CloudSQL service, and the DB in question is configured using CloudSQL's high-availability feature.QueryRowmethod.From what I can tell, all of the request handling goroutines are blocked under this same stack, and there don't seem to be any that are actually holding DB connections.
I've also been able to reproduce this running entirely locally by interposing Toxiproxy in between the application and the DB, and adding a timeout toxic to the proxied connection. Even after this toxic is removed, the application will not recover once it's gotten into this state.
Interestingly, the application does recover if the duration of the toxic being in place is short enough (a few seconds). Even with a high rate of incoming requests (500 reqs/s with a concurrency of 500), it seems to take anywhere from 10 - 60 seconds for it to get stuck like this, suggesting a timing-dependent issue of some kind.
I'm hoping to be able to winnow this down to an easily-shareable test case soon.
Expected behavior
I would expect for the application to be impacted while the DB is unreachable or unresponsive, but that the pool should become usable again once the DB recovers.
Actual behavior
If the DB unreachability goes on for long enough, the pool will eventually enter a state wherein any goroutine that attempts to Acquire a connection hangs indefinitely.
Version
go version go1.19 darwin/arm64PostgreSQL 14.5 (Debian 14.5-1.pgdg110+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit