GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
403 stars 115 forks source link

Multi table validations exhausted PostgreSQL connections #1195

Closed nj1973 closed 2 months ago

nj1973 commented 3 months ago

When running a single validate command for many tables it was noticed that many PostgreSQL connections are opened, the command failed with the error below:

File "/usr/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 598, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "127.0.0.1", port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

I've tested this on PostgreSQL and Oracle and can see the connection count climbing until the command eventually completes and all connections are released.

nj1973 commented 3 months ago

Helper information (based on my DVT username starting with a specific string).

To monitor PostgreSQL connections I used:

SELECT pid,wait_event,state,query FROM pg_stat_activity where usename like 'dvt%';
\watch 1

To monitor Oracle connections I used:

column event format a30
column module format a10 trunc
column action format a10 trunc
set lines 200 tab off pages 100
SELECT sid,event,module,action,state FROM v$session where username like 'DVT%';
nj1973 commented 2 months ago

Since merging this change we are occasionally seeing this warning from DVT:

07/25/2024 09:26:57 AM-WARNING: Exception closing connections: 'Backend' object has no attribute 'con'

We need to make the original fix more robust.