search

celery / django-celery

Old Celery integration project for Django
http://celery.github.com/django-celery
BSD 3-Clause "New" or "Revised" License
1.53k stars 453 forks source link

Celery loosing the connexion with Postgres #413

Open tvanesse opened 8 years ago

tvanesse commented 8 years ago

I have a Django (1.8) application hosted on Heroku and using a Postgres (9.4.4) database. There are 2 pro dynos: one that runs the web server, one that runs a Celery (3.1) worker, Celery being configured with the database backend.

The application has been running smoothly for the past weeks/months but for a couple of days I'm experiencing some downtimes of a few seconds every now and then.

From the logs, it appears to be related to the dynos being cycled for an unknown reason:

...<some successful requests>...
heroku[web.1]: State changed from up to starting
heroku[web.1]: Cycling
heroku[web.1]: Stopping all processes with SIGTERM
... Cycling in progress
heroku[web.1]: State changed from starting to up

and according to the heroku documentation this could be "OK" as dynos need to be cycled at least once a day.

In the meantime the same thing happens for the worker, but this is where the problems come:

heroku[worker.1]: State changed from up to starting
heroku[worker.1]: Cycling
heroku[worker.1]: Stopping all processes with SIGTERM
app[worker.1]: worker: Warm shutdown (MainProcess)
app[worker.1]: [INFO/MainProcess] beat: Shutting down...

...Celery being restarted...

app[worker.1]: [INFO/Beat] beat: Starting...
app[worker.1]: [INFO/MainProcess] Connected to amqp://my-queue-address
app[worker.1]: [INFO/Beat] Writing entries...
app[worker.1]: [INFO/MainProcess] mingle: searching for neighbors
app[worker.1]: [WARNING/MainProcess] celery@xxxxxxxxxxx ready.
app[worker.1]: [INFO/MainProcess] mingle: all alone
app[worker.1]: [INFO/Beat] Scheduler: Sending due task celery.backend_cleanup (celery.backend_cleanup)
app[worker.1]: [INFO/MainProcess] Received task: celery.backend_cleanup[13f89001-08b0-434b-bc7e-372ffcfef521]
app[worker.1]: [INFO/Beat] Writing entries...
app[worker.1]: [INFO/MainProcess] Task celery.backend_cleanup[13f89001-08b0-434b-bc7e-372ffcfef521] succeeded in 0.00162791600451s: None
app[worker.1]: [ERROR/Beat] Database gave error:
OperationalError('terminating connection due to administrator command\nSSL connection has been closed unexpectedly\n',)

...Flood of error logs...

app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
app[worker.1]:     six.reraise(dj_exc_type, dj_exc_value, traceback)
app[worker.1]: InterfaceError: connection already closed

And the problem occurs now more than 15 times a day.

Does anyone know why the dynos are receiving this SIGTERM signal? Also why would cycling lead to the "OperationalError: terminating connection due to administrator command" error? Why is Postgres closing its connections even though we are far below the limit set by heroku? This may or may not be related to django-celery, I have no idea.

kartikdanidhariya commented 8 years ago

hi i am new to heroku app can you please provide detail for deploy django celery in heroku i tried with heroku-redis but fail not return any log and not run task as well my isuue in detail https://github.com/celery/django-celery/issues/455

please give me your valuable suggstion for that

noamgal85 commented 4 years ago

Hi @tvanesse, did you find a solution for this issue? it's been 4 years now and I'm experiencing the exact same thing. Thanks