HikariCP recommends a fixed-size connection pool, so that threads don't have to wait for new connections to be opened during usage spikes. However, with 10 connections per-pod this quickly adds up and is leading to memory exhaustion on the Oracle database.
Each service is typically only using a single connection at a time, with some services spiking to 10 during bulk processing of messages - which lines up with the AWS SQS default of receiving up to 10 messages at a time.
This change means that a single idle connection will always be available for use. As soon as the idle connection is picked up for use from the pool, a new idle connection will be opened to maintain the minimumIdle setting of 1, up until the maximumPoolSize of 10 is reached. If there is more than 1 idle connection, they will be closed after 10 minutes of inactivity until the minimumIdle setting of 1 is reached.
Overall, this means we will be opening and closing connections more often - but the number of open connections at a given time will be much lower than before.
HikariCP recommends a fixed-size connection pool, so that threads don't have to wait for new connections to be opened during usage spikes. However, with 10 connections per-pod this quickly adds up and is leading to memory exhaustion on the Oracle database.
Each service is typically only using a single connection at a time, with some services spiking to 10 during bulk processing of messages - which lines up with the AWS SQS default of receiving up to 10 messages at a time.
The important part of this change is in helm-defaults, here:
PI-2594-reduce-min-pool-size?expand=1#diff-c27a2d56d1This change means that a single idle connection will always be available for use. As soon as the idle connection is picked up for use from the pool, a new idle connection will be opened to maintain the minimumIdle setting of 1, up until the maximumPoolSize of 10 is reached. If there is more than 1 idle connection, they will be closed after 10 minutes of inactivity until the minimumIdle setting of 1 is reached.
Overall, this means we will be opening and closing connections more often - but the number of open connections at a given time will be much lower than before.