k8s db connection errors

Occasionally during heavy loads, metadig-controller generates the following type of errors:

org.postgresql.util.PSQLException: ERROR: query_wait_timeout
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
        at org.postgresql.core.SetupQueryRunner.run(SetupQueryRunner.java:53)
        at org.postgresql.core.v3.ConnectionFactoryImpl.runInitialQueries(ConnectionFactoryImpl.java:674)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:223)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:454)
        at org.postgresql.Driver.connect(Driver.java:256)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at edu.ucsb.nceas.mdqengine.store.DatabaseStore.init(DatabaseStore.java:83)
        at edu.ucsb.nceas.mdqengine.store.DatabaseStore.<init>(DatabaseStore.java:52)
        at edu.ucsb.nceas.mdqengine.store.StoreFactory.getStore(StoreFactory.java:16)

These have to do with the available connections being managed by pgbouncer. Possible causes:

the number of configured pgbouncer connections is not high enough
metadig-engine is not releasing connections -
metadig-engine queries are taking to long

Update the pgbouncer configuration and monitor the system under heavy load, using the pgbouncer admin console to evaluate connection number and duration.

NCEAS / metadig-engine

k8s db connection errors #251