search

rq / django-rq

A simple app that provides django integration for RQ (Redis Queue)
MIT License
1.84k stars 285 forks source link

broken state after redis reset (restart) #562

Open enapupe opened 1 year ago

enapupe commented 1 year ago

hi there, apologies in advance if this is not appropriate or not related to this project itself.

today we had a minor outage caused by a redis restart: all workers kept running after this disconnect (redis went down for a patch upgrade) but seemingly workers did not registered and stopped processing any entries from the queue ever since the restart. killing the it all and starting workers again got things back on track

is that by design? I'm not very familiar with this project TBH, I found very strange that this reconnection wasn't handled gracefully. I'm currently on django-rq = "==2.3.1" rq = "==1.2.2"

should I be specifically handling this situation with a custom exception handler? I don't have any descent logs but I assume the worker has successfully reconnected to redis, it just did not process the queue anymore. thanks in advance

edit: before posting it, I found this https://github.com/rq/rq/pull/1387 would that fix the issue reported above? I'm keeping this issue because I think it might help others facing the same problem.