amqp connections do not close in a timely fashion

FinalsClub / karmaworld

KarmaNotes.org v3.0

GNU Affero General Public License v3.0

7 stars 6 forks source link

amqp connections do not close in a timely fashion #407

Open btbonval opened 9 years ago

btbonval commented 9 years ago

After shutting down client connections to CloudAMQP, I see those connections persist for longer than I have the patience to wait out. It's on the order of minutes.

It looks like the celery worker/beat agents do not cleanly close the upstream connection, and the server doesn't shut it down fast enough for my liking. Maybe it'll close after 10 minutes or a day, but no thanks.

I see there's a timeout field that is always empty. It'd be great if we could somehow push a timeout. I don't see anything in python manage.py celery worker --help or python manage.py celery beat --help. There needs to be some way for us to establish that phantom connections should be closed.

btbonval commented 9 years ago

Some other things to look into as well wrt to celery:

celery worker has a -B flag to run beat concurrently. If this works as --help would suggest, we would not need celerywrapper.sh #415
restrict to the appropriate queue using -Q. as of now it seems like all workers will work on all queues, be they production or staging. not a huge problem, but division of labor is assumed. #413

btbonval commented 9 years ago

If there's a way to shut down a connection when there are no consumers, that'd be stellar. That seems to be the difference. If I'm running celery on my system, the connection will have consumers. If I'm not, the connections remain but there are no consumers.

btbonval commented 9 years ago

I thought fixing #415 by putting celery and beat together as one call in the Procfile might square this problem away.

Unfortunately, closing the foreman executable leaves connections in place according to CloudAMQP. Local connection list doesn't show anything established.