TheRacetrack / racetrack

An opinionated framework for deploying, managing, and serving application workloads
https://theracetrack.github.io/racetrack/
Apache License 2.0
28 stars 5 forks source link

Increase pgbouncer's query_wait_timeout #396

Closed iszulcdeepsense closed 6 months ago

iszulcdeepsense commented 6 months ago

Hubert encountered this:

pgbouncer             | 2023-12-18 18:58:03.882 UTC [1] LOG C-0x7fbcc4faabe0: racetrack/racetrack@172.27.0.8:50574 closing because: query_wait_timeout (age=10s)
pgbouncer             | 2023-12-18 18:58:03.882 UTC [1] WARNING C-0x7fbcc4faabe0: racetrack/racetrack@172.27.0.8:50574 pooler error: query_wait_timeout
lifecycle             | Exception in thread Thread-22 (deploy_job_saving_result):
lifecycle             | Traceback (most recent call last):
lifecycle             |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
lifecycle             |     self.connect()
lifecycle             |   File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
lifecycle             |     return func(*args, **kwargs)
lifecycle             |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 272, in connect
lifecycle             |     self.init_connection_state()
lifecycle             |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 312, in init_connection_state
lifecycle             |     commit_tz = self.ensure_timezone()
lifecycle             |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 294, in ensure_timezone
lifecycle             |     cursor.execute(self.ops.set_time_zone_sql(), [timezone_name])
lifecycle             |   File "/usr/local/lib/python3.10/site-packages/psycopg/cursor.py", line 737, in execute
lifecycle             |     raise ex.with_traceback(None)
lifecycle             | psycopg.OperationalError: consuming input failed: query_wait_timeout
lifecycle             | server closed the connection unexpectedly
lifecycle             |     This probably means the server terminated abnormally
lifecycle             |     before or while processing the request.

This might be due to low query_wait_timeout in combination with a busy CPU