See validate() func in validate stage.
It opens a new connection and only closes it at the end of it's work.
But in single process mode, it calls process_hash_list() and process_hash_list() opens it's own connection!
It needs to do this in multiple process mode, so each process has it's own psql connection.
But in single mode, we have an unused connection that is idle and may timeout.
BUT in validate() function, the connection is not used after getUnvalidatedDatasets is called. Can we close it straight away, before any actual work is done?
This may apply to other stages
Severity
Low - it results in one extra connection per stage so not a big deal, but the unused connections may hang due to idleness (see https://github.com/IATI/refresher/issues/317 )
Notes
See validate() func in validate stage. It opens a new connection and only closes it at the end of it's work.
But in single process mode, it calls process_hash_list() and process_hash_list() opens it's own connection!
It needs to do this in multiple process mode, so each process has it's own psql connection.
But in single mode, we have an unused connection that is idle and may timeout.
BUT in validate() function, the connection is not used after getUnvalidatedDatasets is called. Can we close it straight away, before any actual work is done?
This may apply to other stages
Severity Low - it results in one extra connection per stage so not a big deal, but the unused connections may hang due to idleness (see https://github.com/IATI/refresher/issues/317 )