ERR NOAUTH Authentication required [redis_writer] sentinel=true

alkazap commented 6 months ago

Issue Description

Trying to copy data from standalone Redis to Redis Sentinel, getting ERR NOAUTH Authentication required even though password is specified correctly.

Environment

RedisShake Version：latest
Redis Source Version：latest
Redis Destination Version：latest
Redis 部署方式（standalone/cluster/sentinel）：standalone -> sentinel
Deployed on Cloud Provider：Kubernetes cluster (also tested using Docker Compose)

Logs

2024-05-29 07:59:46 INF load config from file: shake.toml
2024-05-29 07:59:46 INF log_level: [info], log_file: [/usr/src/app/RedisShake/bin/data/shake.log]
2024-05-29 07:59:46 INF changed work dir. dir=[/usr/src/app/RedisShake/bin/data]
2024-05-29 07:59:46 INF GOMAXPROCS defaults to the value of runtime.NumCPU [32]
2024-05-29 07:59:46 INF not set pprof port
2024-05-29 07:59:46 INF create SyncStandaloneReader: dev-redis-replicas-0.dev-redis-headless.dev-redis:6379
2024-05-29 07:59:46 ERR NOAUTH Authentication required.
            RedisShake/internal/client/redis.go:83 -> (*Redis).DoWithStringReply()
            RedisShake/internal/client/redis.go:70 -> NewRedisClient()
            RedisShake/internal/client/redis.go:24 -> NewSentinelClient()
            RedisShake/internal/writer/redis_sentinel_writer.go:11 -> NewRedisSentinelWriter()
            RedisShake/cmd/redis-shake/main.go:104 -> main()
            runtime/proc.go:271 -> main()
            runtime/asm_amd64.s:1695 -> goexit()

Additional Information

shake.toml:

[sync_reader]
address = "dev-redis-master-0.dev-redis-headless.dev-redis:6379"

[redis_writer]
sentinel = true
master = "redis-ha-master"
address = "dev-redis-ha.dev-redis-ha:26379"
password = "password"

I also tried connecting to sentinel master instead:

[sync_reader]
address = "dev-redis-master-0.dev-redis-headless.dev-redis:6379"

[redis_writer]
address = "dev-redis-ha.dev-redis-ha:6379"
password = "password"

But then RedisShake tries to write to read-only replica:

2024-05-29 08:02:13 INF load config from file: shake.toml
2024-05-29 08:02:13 INF log_level: [info], log_file: [/usr/src/app/RedisShake/bin/data/shake.log]
2024-05-29 08:02:13 INF changed work dir. dir=[/usr/src/app/RedisShake/bin/data]
2024-05-29 08:02:13 INF GOMAXPROCS defaults to the value of runtime.NumCPU [32]
2024-05-29 08:02:13 INF not set pprof port
2024-05-29 08:02:13 INF create SyncStandaloneReader: dev-redis-replicas-0.dev-redis-headless.dev-redis:6379
2024-05-29 08:02:13 INF create RedisStandaloneWriter: dev-redis-ha.dev-redis-ha:6379
2024-05-29 08:02:13 INF not set status port
2024-05-29 08:02:13 INF start syncing...
2024-05-29 08:02:13 ERR [writer_dev-redis-ha.dev-redis-ha_6379] receive reply failed. cmd=[set image:async:input:c236c419-936e-4d9b-87e2-cd5eeec7c045 {"image":{"source":{"url":"https://storag...], error=[READONLY You can't write against a read only replica.]
            RedisShake/internal/writer/redis_standalone_writer.go:116 -> (*redisStandaloneWriter).processReply()
            runtime/asm_amd64.s:1695 -> goexit()

suxb201 commented 6 months ago

You have also configured a password for the sentinel nodes. The password specified in the shake.toml will only be applied to the redis data nodes, as by default, it is assumed that the sentinel nodes do not have a password configured. You may need to modify the code for it to take effect, or alternatively, you can temporarily use the standalone writer instead of the sentinel writer. https://github.com/tair-opensource/RedisShake/pull/806
I suspect that dev-redis-ha.dev-redis-ha:6379 returned a read-only replica node. You can test this using redis-cli.

alkazap commented 6 months ago

Right we configured password for the sentinel, is that not the common practice?

~Unfortunately I'm not very familiar with your code base and golang, could you please point me to a file where I should modify the code?~ [Edit] Figured it out by looking at the linked feature.

You're right, read-only replica was returned, I could try to check which node is the master before running RedisShake...

Can RedisShake handle failover scenario if connected to Sentinel? [Edit] After looking at the linked feature seems like it doesn't.

alkazap commented 6 months ago

Thank you for your help. I modified the code and now it works. We set sentinel password and master password to be the same. Not sure if this is common practice. In case it is, I can submit it as feature.

tair-opensource / RedisShake