This issue is most prevalent in dFlow due to the number of agents contained in the project, however the cause was identified during a debugging session with Ontario's onbis-x-agent were multiple instances were being spun up to perform an initial data load.
The issue occurs when there are a number of agents spinning up at the same time and connecting to the same wallet. The number of connections opened by the multiple agents exceeds the max connections of the database. New connections then fail, with the stated error, until previous ones are released. The other side affect of this is that the health check on the database container starts to fail because it can't connect due to the lack of available connections. As a result the database container will stop receiving requests, altogether, for a period of time.
Before we just go upping the number of connections on the wallet database we should have a look into the postgresql wallet plug-in's connection management.
This issue is most prevalent in dFlow due to the number of agents contained in the project, however the cause was identified during a debugging session with Ontario's
onbis-x-agentwere multiple instances were being spun up to perform an initial data load.The issue occurs when there are a number of agents spinning up at the same time and connecting to the same wallet. The number of connections opened by the multiple agents exceeds the max connections of the database. New connections then fail, with the stated error, until previous ones are released. The other side affect of this is that the health check on the database container starts to fail because it can't connect due to the lack of available connections. As a result the database container will stop receiving requests, altogether, for a period of time.
Before we just go upping the number of connections on the wallet database we should have a look into the postgresql wallet plug-in's connection management.