This is a further attempt to address the celery error which Sebastian still encounters. I think of 4 possible causes for this error:
Celery is not running before running a tuning task (this type of error is fixed in the previous PR)
Celery dies when it is running a tuning task (this type of error is not considered in the previous PR)
For some reason, Celery can never start.
Celery is running, but rabbitmq server is not running, so Celery cannot work
I fixed the second kind of error in this PR:
set CELERY_ACKS_LATE=True, so that an incomplete task is still in the queue for retrying. Without this option, when celery is killed, the running task is lost forever.
every time the client pings the server for results, I check the Celery status and restart Celery if it is down.
I also examine the fourth kind of error. I have added code to test if rabbitmq is running by connecting the port 5672. We get a warning if it is not running.
Tests:
I inserted codes in a few positions in async_task.py to kill Celery. Without the fixes in this PR the task fails. With the fixes here the task succeeds.
However, I am worried that Sebastian's error is more like the third kind, because there are error message as follows:
Failed to start 'celery'. Failed to start 'celerybeat'.
It means there is no pid file for celery worker and celerybeat. I do not really know what can be a potential cause for that. At least when rabbitmq is not running, celery can still come up and generate the pid files.
The only reason I can think of is that the waiting time is shorter than needed. So I increased it from 15 seconds to 60 seconds, and I log the waiting time to see if it is long or not.
This is a further attempt to address the celery error which Sebastian still encounters. I think of 4 possible causes for this error:
I fixed the second kind of error in this PR:
CELERY_ACKS_LATE=True, so that an incomplete task is still in the queue for retrying. Without this option, when celery is killed, the running task is lost forever.Tests: I inserted codes in a few positions in async_task.py to kill Celery. Without the fixes in this PR the task fails. With the fixes here the task succeeds.
However, I am worried that Sebastian's error is more like the third kind, because there are error message as follows:
Failed to start 'celery'. Failed to start 'celerybeat'.It means there is no pid file for celery worker and celerybeat. I do not really know what can be a potential cause for that. At least when rabbitmq is not running, celery can still come up and generate the pid files. The only reason I can think of is that the waiting time is shorter than needed. So I increased it from 15 seconds to 60 seconds, and I log the waiting time to see if it is long or not.