tair-opensource / RedisShake

RedisShake is a Redis data processing and migration tool.
https://tair-opensource.github.io/RedisShake/
MIT License
3.88k stars 702 forks source link

redis 4.0.14 同步到 6.2.7 数据不一致 #791

Closed stereon closed 6 months ago

stereon commented 7 months ago

问题描述(Issue Description)

使用redis-shake 4.0 数据同步后结果不一致

Please provide a brief description of the issue you encountered.

环境信息(Environment)

日志信息(Logs)

{"level":"info","time":"2024-04-16T08:02:24Z","message":"GOMAXPROCS defaults to the value of runtime.NumCPU [2]"} {"level":"info","time":"2024-04-16T08:02:24Z","message":"not set pprof port"} {"level":"info","time":"2024-04-16T08:02:24Z","message":"create SyncStandaloneReader: 10.18.1.65:6379"} {"level":"info","time":"2024-04-16T08:02:24Z","message":"create RedisStandaloneWriter: 10.18.47.235:6379"} {"level":"info","time":"2024-04-16T08:02:24Z","message":"not set status port"} {"level":"info","time":"2024-04-16T08:02:24Z","message":"start syncing..."} {"level":"info","time":"2024-04-16T08:02:29Z","message":"read_count=[4651], read_ops=[215.99], write_count=[4651], write_ops=[215.99], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:02:34Z","message":"read_count=[5550], read_ops=[164.11], write_count=[5550], write_ops=[164.11], syncing aof, diff=[572]"} {"level":"info","time":"2024-04-16T08:02:39Z","message":"read_count=[6821], read_ops=[253.14], write_count=[6821], write_ops=[253.14], syncing aof, diff=[143]"} {"level":"info","time":"2024-04-16T08:02:44Z","message":"read_count=[8137], read_ops=[267.13], write_count=[8137], write_ops=[267.13], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:02:49Z","message":"read_count=[9144], read_ops=[219.30], write_count=[9144], write_ops=[219.30], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:02:54Z","message":"read_count=[10198], read_ops=[204.05], write_count=[10198], write_ops=[204.05], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:02:59Z","message":"read_count=[11219], read_ops=[145.02], write_count=[11219], write_ops=[145.02], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:03:04Z","message":"read_count=[12176], read_ops=[166.97], write_count=[12176], write_ops=[166.97], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:03:09Z","message":"read_count=[13238], read_ops=[166.20], write_count=[13238], write_ops=[166.20], syncing aof, diff=[286]"} {"level":"info","time":"2024-04-16T08:03:14Z","message":"read_count=[14115], read_ops=[176.94], write_count=[14115], write_ops=[176.94], syncing aof, diff=[143]"} {"level":"info","time":"2024-04-16T08:03:19Z","message":"read_count=[15086], read_ops=[203.44], write_count=[15086], write_ops=[203.44], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:03:24Z","message":"read_count=[16175], read_ops=[190.88], write_count=[16175], write_ops=[190.88], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:03:29Z","message":"read_count=[17111], read_ops=[206.12], write_count=[17111], write_ops=[206.12], syncing aof, diff=[143]"} {"level":"info","time":"2024-04-16T08:03:34Z","message":"read_count=[18117], read_ops=[209.07], write_count=[18117], write_ops=[209.07], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:03:39Z","message":"read_count=[19063], read_ops=[199.13], write_count=[19063], write_ops=[199.13], syncing aof, diff=[0]"} {"level":"info","time":"2024-04-16T08:03:44Z","message":"read_count=[20097], read_ops=[166.03], write_count=[20097], write_ops=[166.03], syncing aof, diff=[0]"}

数据对比结果: 新实例:# Keyspace db0:keys=1221,expires=1194,avg_ttl=116766

旧实例: # Keyspace db0:keys=1244,expires=1217,avg_ttl=114422

suxb201 commented 7 months ago

1221-1194=27 1244-1217=27 数据一致

suxb201 commented 7 months ago

我猜是你查看 info 的时间不一样,期间发生了过期。你可以找几个 key 看看过期时间。

stereon commented 7 months ago

1221-1194=27 1244-1217=27 数据一致 这里的 keys 总数为什么不是强一致的

stereon commented 7 months ago

我猜是你查看 info 的时间不一样,期间发生了过期。你可以找几个 key 看看过期时间。

这里写的脚本同时去查的源端和目标端,按理说应该是一致的,我这里可以在测试一下,目前源端是没有写入的,key 的数量是不变的

stereon commented 7 months ago

另外有拿了一套redis 实例,发现key 还是不一样 源端:# Keyspace db0:keys=28149,expires=28149,avg_ttl=221134947 目标端:# Keyspace db0:keys=25361,expires=25361,avg_ttl=223395805

suxb201 commented 7 months ago

@stereon 感觉和过期有关系,你可以找到具体有差异的 key,然后看一下过期时间吗?

suxb201 commented 6 months ago

提供文档说明。