Postgres allows 100 maximum simultaneous connections in our current configuration, which should be more than enough for our own purposes.
However we haven't been using them effectively (giving a connection pool size of 10 to each of mqtt-task, auth, api and sidekiq), leading to errors in sidekiq jobs - as sidekiq 25 worker threads in production.
I've upped this to 25 per container (since there are 4), which in theory should have solved the sidekiq problem (25 threads, 25 connections, should be one each). However we're still seeing occasional errors: I assume that sidekiq also has some sort of manager thread which itself can use a connection.
I've temporarily decreased the number of sidekiq worker threads in production to 20, which seems to have stopped the errors. However, this isn't ideal, most notably because mqtt-task is single threaded and therefore has 24 connections more than it actually needs.
A better solution would be to provide an appropriately sized connection pool to each service, however, the pool size is currently set by an env variable in the shared env file. A better approach would be to set this per-service in the docker compose file.
This ticket is to do this, as well as to research and set appropriate database connection pool sizes for each service.
Postgres allows 100 maximum simultaneous connections in our current configuration, which should be more than enough for our own purposes.
However we haven't been using them effectively (giving a connection pool size of 10 to each of mqtt-task, auth, api and sidekiq), leading to errors in sidekiq jobs - as sidekiq 25 worker threads in production.
I've upped this to 25 per container (since there are 4), which in theory should have solved the sidekiq problem (25 threads, 25 connections, should be one each). However we're still seeing occasional errors: I assume that sidekiq also has some sort of manager thread which itself can use a connection.
I've temporarily decreased the number of sidekiq worker threads in production to 20, which seems to have stopped the errors. However, this isn't ideal, most notably because mqtt-task is single threaded and therefore has 24 connections more than it actually needs.
A better solution would be to provide an appropriately sized connection pool to each service, however, the pool size is currently set by an env variable in the shared env file. A better approach would be to set this per-service in the docker compose file.
This ticket is to do this, as well as to research and set appropriate database connection pool sizes for each service.