search

GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
400 stars 114 forks source link

fix: Close source and target connections after executing a validation #1197

Closed nj1973 closed 2 months ago

nj1973 commented 2 months ago

When multiple tables are validated in a single command we are building up a number of idle connections, at least in Oracle and PostgreSQL we are. Example command where ${TABLES} contains find-tables output of 23 tables:

data-validation  validate column -sc=ora_local -tc=pg_local -tbls="${TABLES}"

When I checked Oracle & PostgreSQL during the command we accumulated > 12 idle connections.

I changed the DataValidation class to support being called from a context manager and explicitly closed connections on exit. After this my test command never had more than two connections open at any time.

I also added some code to Oracle client to tag database sessions with "DVT" string.

nj1973 commented 2 months ago

/gcbrun

helensilva14 commented 2 months ago

/gcbrun