Postgres server connections filling up

[mikkonie]

On the production deployment of SODAR, we have had an issue where the postgres server suddenly crashes with "too many clients". This has happened sporadically, a few times within a month.

I've also observed this suddenly happening on my development environment. It would seem something is keeping connections open. I'm not sure whether it is SODAR or iRODS. When I saw this in my dev environment I had just launched SODAR for the 1st time in a while, so my guess would be iRODS which has been running in the background.

Some steps to get further with this:

• Check for active connections
  • SELECT datname, client_addr, query_start, state_change, state FROM pg_stat_activity ORDER BY query_start;
  • https://www.postgresql.org/docs/current/monitoring-stats.html
  • docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' sodar-docker-compose_irods_1
• Increase max connections?
• Set conn_max_age = 0?

To be investigated further.

[mikkonie]

Looking at pg_stat_activity during normal operation, I'm seeing a bunch of idle processes for ICAT originating from the iRODS host. Might be normal or not, can't tell.

It's not too many of these processes at the moment, but it's Monday morning and it's possible these accumulate over time.

I'm purely speculating, but maybe these iRODS session disconnection problems I've been observing at bihealth/sodar-server#1542 are related to these idle postgres connections and that's what's causing this problem?

Will try to monitor the situation, also attempting to reproduce this again locally in my dev environment.

[mikkonie]

After some more investigation, it turns out this is indeed caused by SODAR behaviour. Attempting a fix on that side, will keep this issue open until I've confirmed the problem is gone.

[mikkonie]

This was fixed by bihealth/sodar-server#909, closing.