ERR The ID specified in XADD is equal or smaller than the target stream top item

alkazap commented 5 months ago

Issue Description

RedisShake errors out whenever it receives XADD command

Environment

RedisShake Version：latest
Redis Source Version：latest
Redis Destination Version：latest
Redis 部署方式（standalone/cluster/sentinel）：standalone -> sentinel
Deployed on Cloud Provider：Kubernetes cluster (also tested using Docker Compose)

Logs

2024-05-29 08:07:18 INF load config from file: shake.toml
2024-05-29 08:07:18 INF log_level: [info], log_file: [/usr/src/app/RedisShake/bin/data/shake.log]
2024-05-29 08:07:18 INF changed work dir. dir=[/usr/src/app/RedisShake/bin/data]
2024-05-29 08:07:18 INF GOMAXPROCS defaults to the value of runtime.NumCPU [32]
2024-05-29 08:07:18 INF not set pprof port
2024-05-29 08:07:18 INF create SyncStandaloneReader: dev-redis-replicas-0.dev-redis-headless.dev-redis:6379
2024-05-29 08:07:18 INF create RedisStandaloneWriter: dev-redis-ha.dev-redis-ha:6379
2024-05-29 08:07:18 INF not set status port
2024-05-29 08:07:18 INF start syncing...
2024-05-29 08:07:18 ERR [writer_dev-redis-ha.dev-redis-ha_6379] receive reply failed. cmd=[xadd nlp:sync:output:4bcce72b-cb24-4b54-949e-867a9f769364:stream 1709602510226-0 debug {"nlp_summary...], error=[ERR The ID specified in XADD is equal or smaller than the target stream top item]
            RedisShake/internal/writer/redis_standalone_writer.go:116 -> (*redisStandaloneWriter).processReply()
            runtime/asm_amd64.s:1695 -> goexit()

suxb201 commented 5 months ago

Could you check if there are any xadd operations being performed on the destination Redis instance? These operations might be causing the issue you’re encountering.

alkazap commented 5 months ago

~right there are XADD operations being performed. Does it mean that RedisShake can't sync those?~

alkazap commented 5 months ago

my bad, I did not read your question carefully. XADD operations are only performed on the source Redis, not destination. I found that you create Redis Stream with "0-1" https://github.com/tair-opensource/RedisShake/pull/808. I wonder if that is causing this issue.

alkazap commented 5 months ago

I am first performing XREAD command with "0-0" and then XADD uses appended ID "0-1" (not set by default)

Sometimes XADD happens before XREAD

suxb201 commented 5 months ago

I can't think of a reason for now.

alkazap commented 5 months ago

So I went ahead and deleted the stream from src and dest, restarted RedisShake and error changed:

2024-05-30 02:39:52 INF load config from file: shake.toml
2024-05-30 02:39:52 INF log_level: [info], log_file: [/usr/src/app/RedisShake/bin/data/shake.log]
2024-05-30 02:39:52 INF changed work dir. dir=[/usr/src/app/RedisShake/bin/data]
2024-05-30 02:39:52 INF GOMAXPROCS defaults to the value of runtime.NumCPU [32]
2024-05-30 02:39:52 INF not set pprof port
2024-05-30 02:39:52 INF create SyncStandaloneReader: dev-redis-replicas-0.dev-redis-headless.dev-redis:6379
2024-05-30 02:39:52 INF connecting to master node at dev-redis-ha-node-0.dev-redis-ha-headless.dev-redis-ha.svc.cluster.local:6379
2024-05-30 02:39:52 INF create RedisSentinelWriter: dev-redis-ha.dev-redis-ha:26379
2024-05-30 02:39:52 INF not set status port
2024-05-30 02:39:52 INF start syncing...
2024-05-30 02:39:52 ERR [writer_dev-redis-ha-node-0.dev-redis-ha-headless.dev-redis-ha.svc.cluster.local_6379] receive reply failed. cmd=[xadd nlp:sync:output:4bcce72b-cb24-4b54-949e-867a9f769364:stream MAXLEN 0 0-1 x y], error=[ERR The ID specified in XADD is equal or smaller than the target stream top item]
            RedisShake/internal/writer/redis_standalone_writer.go:116 -> (*redisStandaloneWriter).processReply()
            runtime/asm_amd64.s:1695 -> goexit()

before: receive reply failed. cmd=[xadd nlp:sync:output:4bcce72b-cb24-4b54-949e-867a9f769364:stream 1709602510226-0

after: receive reply failed. cmd=[xadd nlp:sync:output:4bcce72b-cb24-4b54-949e-867a9f769364:stream MAXLEN 0 0-1 x y]

suxb201 commented 5 months ago

Try using the function to filter all stream class commands. https://tair-opensource.github.io/RedisShake/en/function/best_practices.html#%E8%BF%87%E6%BB%A4%E6%9F%90%E7%B1%BB%E6%95%B0%E6%8D%AE%E7%BB%93%E6%9E%84

alkazap commented 5 months ago

Thank you, that's a good temporary fix

tair-opensource / RedisShake