Open lifenautjoe opened 4 years ago
Something that might be is that there was a redeploy? We have a script which post deploy restarts supervisord with the custom config including the djangorq program.
Perhaps this doesn't get run in all situations?
files:
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/04_update_supervisor.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf -s unix:///opt/python/run/supervisor.sock reload
Nope, there was no redeploy at that time. On the 28th of August was the last one. There is no error in the logging whatsoever.
Today we had a queue of 500+ jobs not being processed by rq workers.
The second machine was running a couple worker processes.
But supervisor wasn't aware of a running program:rqworker. I tried to stop the service and got
Once I started it
Jobs started to process again.
Perhaps related to https://github.com/rq/rq/issues/758
Is our supervisor config for rqworker correct?
We MUST ensure that this doesn't happen again as we will now use rq workers to process post media.
If they stall, no posting will be possible.
@evict halp