search

hibiken / asynq

Simple, reliable, and efficient distributed task queue in Go
MIT License
10.03k stars 716 forks source link

[BUG] asynq can not follow the right redis master when using sentinel mode. #959

Open hoorayman opened 2 weeks ago

hoorayman commented 2 weeks ago

Describe the bug asynq can not follow the right redis master.

Environment:

To Reproduce use helm to install a redis cluster, with sentinel be set to true. configure the asynq using sentinel: image

Expected behavior asynq can follow the right master and running normally.

Screenshots go-redis can find the right master and change: image But asynq report error: asynq: pid=1 2024/11/07 08:23:58.539202 ERROR: Dequeue error: UNKNOWN: redis eval error: READONLY You can't write against a read only replica. script: a8153319360adc71cdc370107f8ab9786f204765, on @user_script:3. asynq: pid=1 2024/11/07 08:23:58.557608 ERROR: Failed to write server state data: UNKNOWN: redis command error: SADD failed: READONLY You can't write against a read only replica.

kamikazechaser commented 1 week ago

This is most likely an issue with misconfiguration when using go-redis with Redis sentinel (Also, because you are running this in k8s, you will most likely have only 1 service that points to the redis sentinel endpoint). Could you debug a simple go-redis application first https://redis.uptrace.dev/guide/go-redis-sentinel.html#redis-server-client?