Out of connections on postgresql

[macobo]

Sentry error: https://sentry.io/organizations/posthog/issues/2225546229/?project=5592816&query=is%3Aunassigned+is%3Aunresolved

[___3] 13:22:35 🟡 Can not start job queue producer "graphile": remaining connection slots are reserved for non-replication superuser connections
error Parser.parseErrorMessage(pg-protocol.dist:parser)
remaining connection slots are reserved for non-replication superuser connections

This might be because either:

• graphile worker got introduced
• pgbouncer limits

Worth looking into:

• What amount of connections is graphile trying to open
• Are we running the graphile DB behind a pgbouncer?

[macobo]

Seems like we're running Aurora Postgresql directly on RDS, with no pgbouncer.

Here you can see our connection usage over last week:

While there are connection limits on it, it seems we've set it to autoscale. Here's the "number of instances" count over the last week:

Looking at other metrics, seems like we're doing very little traffic through it. Seems like we might need to tweak this instance setup?

cc @fuziontech - mind documenting the existing setup and handing off the "fix" to tiina or me?

[fuziontech]

Fixed! We added a PGBouncer service between Aurora and Graphile https://github.com/PostHog/private-infra/tree/main/aws/infra/services/pgq-pgbouncer