hibiken / asynq

Simple, reliable, and efficient distributed task queue in Go
MIT License
9.49k stars 688 forks source link

[BUG] After manually deleted redis keys, some of the features like metrics, inspector aren't working properly #901

Open sailesh2341 opened 2 months ago

sailesh2341 commented 2 months ago

Describe the bug I have manually deleted one of the key from redis DB to remove one of the task under archive state from queue. After doing this asynqmon metrics stopped working and Inspector methods aren't working to get info of tasks/queue. The error I'm getting both for metrics and Inspector methods are as follows:

asynq UNKNOWN: UNKNOWN: redis eval error: ERR Error running script (call to f_0c80fc1868e3aae205155ae2017ce995a3a7cec0): @user_script:30: user_script:30: attempt to perform arithmetic on local 'bytes' (a boolean value)

To Reproduce I've tried many ways to resolve this error:

  1. Restarted the application to restart the asynq server and other instances.
  2. Tried to manually insert the deleted key into redis DB again.
  3. Tried moving some task to archive state and back to pending state again so that the lost keys can be recreated.

Nothing of these approaches worked. Since this issue was on production environment , i cannot flush all the redis DB data nor i cannot take any risky actions . Has anyone else faced the same issue? Can someone please tell me how do we resolve this issue?