DB Connection leak?

[nkelner]

@Rocket-Listings/owners I'm throwing an issue up here so we don't forget about this. Basically what happened was I tried to run some heroku run manage.py commands that need db connection and I got the error psql: FATAL: too many connections for role "faljgqnmewucar". When I tried to run heroku pg:ps to do some debugging I couldn't even get a connection to see or kill the current connections!

The immediate cause of this error has to do with the fact that our heroku postgres tier only allows 20 connections and the settings for django_postgrespool:

DATABASE_POOL_ARGS = {
  'max_overflow': 10,
  'pool_size': 10,
  'recycle': 300
}

will consume all connections at max overflow levels.

The future problem (although it doesn't affect end users right now) is that even after waiting a considerable time those connections were not being released. This suggests either:

• A connection leak.
• Some poor DB querying in our code.

For now all I'm doing is reducing the max_overflow of our pool to 9 so that there is always a free connections for the heroku pg command to use. That way if this happens again we will at least be able to investigate

[nkelner]

I guess I should mention that obviously a higher herokupg tier would solve this but that would be expensive/may just plaster over a larger issue.

[teddyknox]

Wow, that's a pretty big bug. It's probably slowing the site down a lot too. I'm going to read up on django db connections in relation to gunicorn.

[nkelner]

@teddyknox I brought up the logs for the rocket app and I got this:

If you look at those you can see that the new relic python agent crashed and was trying to make tons connections to the db. I can't be sure that this is the issue but if the connection leak (or whatever it is) appears again, new relic should be the first thing we check. For future reference the way I restarted the new relic agent was heroku run newrelic-admin validate-config --app rocket-listings - stdout

[nkelner]

Also I'm gonna continue to leave this issue open till we're sure we've to the root cause.

[nkelner]

Pretty sure we fixed this.