search

tair-opensource / RedisShake

RedisShake is a Redis data processing and migration tool.
https://tair-opensource.github.io/RedisShake/
MIT License
3.86k stars 700 forks source link

同步数据较慢 #799

Closed cataglyphis closed 6 months ago

cataglyphis commented 6 months ago

问题描述（Issue Description）

同步自建的 redis cluster 数据到 AWS ElastiCache，redis 是 3 主 3 从的配置。 有 2 个节点的 Key 数量在 1200W 左右，有 1 个在 100W 左右

环境信息（Environment）

日志信息（Logs）

2024-05-15 06:57:24 INF create RedisClusterWriter: clustercfg.redis-cluster-prod-us.xnwejy.usw1.cache.amazonaws.com:6379
2024-05-15 06:57:24 INF not set status port
2024-05-15 06:57:24 INF start syncing...
2024-05-15 06:57:29 INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-1, waiting bgsave
2024-05-15 06:57:34 INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-2, waiting bgsave
2024-05-15 06:57:39 INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], src-0, receiving rdb
2024-05-15 06:57:44 INF read_count=[49704], read_ops=[49717.07], write_count=[49703], write_ops=[49716.07], src-1, waiting bgsave
2024-05-15 06:57:49 INF read_count=[350387], read_ops=[58625.47], write_count=[350386], write_ops=[58625.47], src-2, waiting bgsave
2024-05-15 06:57:54 INF read_count=[654089], read_ops=[60445.45], write_count=[654088], write_ops=[60445.45], src-0, syncing rdb, size=[3.9 MiB/492 MiB]
2024-05-15 06:57:59 INF read_count=[959948], read_ops=[61315.85], write_count=[959947], write_ops=[61315.85], src-1, waiting bgsave
2024-05-15 06:58:04 INF read_count=[1259413], read_ops=[59568.26], write_count=[1259412], write_ops=[59569.26], src-2, waiting bgsave
2024-05-15 06:58:09 INF read_count=[1564152], read_ops=[61100.45], write_count=[1564151], write_ops=[61100.45], src-0, syncing rdb, size=[9.6 MiB/492 MiB]
2024-05-15 06:58:14 INF read_count=[1874758], read_ops=[62724.37], write_count=[1874757], write_ops=[62724.37], src-1, waiting bgsave
2024-05-15 06:58:19 INF read_count=[2177452], read_ops=[58886.78], write_count=[2177451], write_ops=[58886.78], src-2, waiting bgsave
2024-05-15 06:58:24 INF read_count=[2485751], read_ops=[63225.79], write_count=[2485750], write_ops=[63224.79], src-0, syncing rdb, size=[16 MiB/492 MiB]
2024-05-15 06:58:29 INF read_count=[2793423], read_ops=[61891.68], write_count=[2793423], write_ops=[61891.68], src-1, waiting bgsave
2024-05-15 06:58:34 INF read_count=[3099020], read_ops=[59701.81], write_count=[3099020], write_ops=[59700.81], src-2, waiting bgsave
2024-05-15 06:58:39 INF read_count=[3400906], read_ops=[58369.00], write_count=[3400905], write_ops=[58369.00], src-0, syncing rdb, size=[21 MiB/492 MiB]
2024-05-15 06:58:44 INF read_count=[3707526], read_ops=[62169.54], write_count=[3707525], write_ops=[62169.54], src-1, waiting bgsave
2024-05-15 06:58:49 INF read_count=[4012763], read_ops=[59020.97], write_count=[4012762], write_ops=[59019.97], src-2, waiting bgsave
2024-05-15 06:58:54 INF read_count=[4316515], read_ops=[60600.20], write_count=[4316514], write_ops=[60600.20], src-0, syncing rdb, size=[27 MiB/492 MiB]
2024-05-15 06:58:59 INF read_count=[4619623], read_ops=[60846.64], write_count=[4619622], write_ops=[60846.64], src-1, waiting bgsave
2024-05-15 06:59:04 INF read_count=[4919546], read_ops=[59948.93], write_count=[4919545], write_ops=[59948.93], src-2, waiting bgsave
2024-05-15 06:59:09 INF read_count=[5213744], read_ops=[61345.92], write_count=[5213743], write_ops=[61345.92], src-0, syncing rdb, size=[33 MiB/492 MiB]
2024-05-15 06:59:14 INF read_count=[5523272], read_ops=[62902.64], write_count=[5523272], write_ops=[62903.64], src-1, receiving rdb
2024-05-15 06:59:19 INF read_count=[5826432], read_ops=[59643.55], write_count=[5826431], write_ops=[59642.55], src-2, receiving rdb
2024-05-15 06:59:24 INF read_count=[6129314], read_ops=[59554.78], write_count=[6129314], write_ops=[59554.78], src-0, syncing rdb, size=[39 MiB/492 MiB]
2024-05-15 06:59:29 INF read_count=[6437981], read_ops=[62391.95], write_count=[6437981], write_ops=[62391.95], src-1, receiving rdb
2024-05-15 06:59:34 INF read_count=[6742262], read_ops=[57873.70], write_count=[6742261], write_ops=[57873.70], src-2, receiving rdb
2024-05-15 06:59:39 INF read_count=[7053437], read_ops=[64293.44], write_count=[7053436], write_ops=[64293.44], src-0, syncing rdb, size=[45 MiB/492 MiB]
2024-05-15 06:59:44 INF read_count=[7353244], read_ops=[58878.97], write_count=[7353243], write_ops=[58878.97], src-1, receiving rdb
2024-05-15 06:59:49 INF read_count=[7660840], read_ops=[61090.99], write_count=[7660839], write_ops=[61091.99], src-2, receiving rdb
2024-05-15 06:59:54 INF read_count=[7965915], read_ops=[63200.30], write_count=[7965915], write_ops=[63200.30], src-0, syncing rdb, size=[51 MiB/492 MiB]
2024-05-15 06:59:59 INF read_count=[8273314], read_ops=[63063.86], write_count=[8273314], write_ops=[63063.86], src-1, receiving rdb
2024-05-15 07:00:04 INF read_count=[8577411], read_ops=[60231.23], write_count=[8577410], write_ops=[60230.23], src-2, receiving rdb
2024-05-15 07:00:09 INF read_count=[8862772], read_ops=[63702.26], write_count=[8862771], write_ops=[63701.26], src-0, syncing rdb, size=[56 MiB/492 MiB]
2024-05-15 07:00:14 INF read_count=[9172598], read_ops=[61363.63], write_count=[9172598], write_ops=[61364.63], src-1, receiving rdb
2024-05-15 07:00:19 INF read_count=[9480032], read_ops=[62779.04], write_count=[9480031], write_ops=[62779.04], src-2, receiving rdb
2024-05-15 07:00:24 INF read_count=[9790131], read_ops=[63962.91], write_count=[9790130], write_ops=[63962.91], src-0, syncing rdb, size=[62 MiB/492 MiB]
2024-05-15 07:00:29 INF read_count=[10100890], read_ops=[62255.06], write_count=[10100889], write_ops=[62255.06], src-1, receiving rdb
2024-05-15 07:00:34 INF read_count=[10405061], read_ops=[59681.40], write_count=[10405060], write_ops=[59681.40], src-2, receiving rdb
2024-05-15 07:00:39 INF read_count=[10713382], read_ops=[61506.67], write_count=[10713381], write_ops=[61507.67], src-0, syncing rdb, size=[68 MiB/492 MiB]
2024-05-15 07:00:44 INF read_count=[11012457], read_ops=[58501.53], write_count=[11012456], write_ops=[58500.53], src-1, receiving rdb
2024-05-15 07:00:49 INF read_count=[11321879], read_ops=[61335.88], write_count=[11321878], write_ops=[61335.88], src-2, receiving rdb
2024-05-15 07:00:54 INF read_count=[11626385], read_ops=[61502.21], write_count=[11626384], write_ops=[61502.21], src-0, syncing rdb, size=[74 MiB/492 MiB]
2024-05-15 07:00:59 INF read_count=[11935964], read_ops=[61171.01], write_count=[11935963], write_ops=[61172.01], src-1, receiving rdb
2024-05-15 07:01:04 INF read_count=[12241517], read_ops=[59477.20], write_count=[12241516], write_ops=[59477.20], src-2, receiving rdb
2024-05-15 07:01:09 INF read_count=[12530486], read_ops=[56598.57], write_count=[12530485], write_ops=[56598.57], src-0, syncing rdb, size=[79 MiB/492 MiB]
2024-05-15 07:01:14 INF read_count=[12818118], read_ops=[56980.89], write_count=[12818118], write_ops=[56980.89], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:01:19 INF read_count=[13081142], read_ops=[48202.00], write_count=[13081141], write_ops=[48202.00], src-2, syncing rdb, size=[368 KiB/5.4 GiB]
2024-05-15 07:01:24 INF read_count=[13336656], read_ops=[52202.20], write_count=[13336655], write_ops=[52202.20], src-0, syncing rdb, size=[81 MiB/492 MiB]
2024-05-15 07:01:29 INF read_count=[13592171], read_ops=[52341.70], write_count=[13592170], write_ops=[52341.70], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:01:34 INF read_count=[13852056], read_ops=[52168.14], write_count=[13852055], write_ops=[52167.14], src-2, syncing rdb, size=[2.5 MiB/5.4 GiB]
2024-05-15 07:01:39 INF read_count=[14114527], read_ops=[52949.90], write_count=[14114526], write_ops=[52949.90], src-0, syncing rdb, size=[83 MiB/492 MiB]
2024-05-15 07:01:44 INF read_count=[14366021], read_ops=[49612.24], write_count=[14366021], write_ops=[49611.24], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:01:49 INF read_count=[14625717], read_ops=[52057.25], write_count=[14625716], write_ops=[52056.25], src-2, syncing rdb, size=[4.0 MiB/5.4 GiB]
2024-05-15 07:01:54 INF read_count=[14878159], read_ops=[50054.16], write_count=[14878158], write_ops=[50054.16], src-0, syncing rdb, size=[84 MiB/492 MiB]
2024-05-15 07:01:59 INF read_count=[15139508], read_ops=[53386.96], write_count=[15139507], write_ops=[53386.96], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:02:04 INF read_count=[15395287], read_ops=[51361.67], write_count=[15395286], write_ops=[51361.67], src-2, syncing rdb, size=[6.1 MiB/5.4 GiB]
2024-05-15 07:02:09 INF read_count=[15657893], read_ops=[52952.58], write_count=[15657893], write_ops=[52952.58], src-0, syncing rdb, size=[86 MiB/492 MiB]
2024-05-15 07:02:14 INF read_count=[15918024], read_ops=[51670.44], write_count=[15918023], write_ops=[51670.44], src-1, syncing rdb, size=[1.6 MiB/5.4 GiB]
2024-05-15 07:02:19 INF read_count=[16173454], read_ops=[49914.77], write_count=[16173453], write_ops=[49914.77], src-2, syncing rdb, size=[8.2 MiB/5.4 GiB]
2024-05-15 07:02:24 INF read_count=[16424770], read_ops=[45679.81], write_count=[16424769], write_ops=[45679.81], src-0, syncing rdb, size=[87 MiB/492 MiB]
2024-05-15 07:02:29 INF read_count=[16685182], read_ops=[52144.54], write_count=[16685181], write_ops=[52144.54], src-1, syncing rdb, size=[8.0 MiB/5.4 GiB]
2024-05-15 07:02:34 INF read_count=[16942793], read_ops=[52668.01], write_count=[16942792], write_ops=[52668.01], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:02:39 INF read_count=[17204845], read_ops=[52398.31], write_count=[17204844], write_ops=[52398.31], src-0, syncing rdb, size=[89 MiB/492 MiB]
2024-05-15 07:02:44 INF read_count=[17460081], read_ops=[49617.08], write_count=[17460080], write_ops=[49617.08], src-1, syncing rdb, size=[10 MiB/5.4 GiB]
2024-05-15 07:02:49 INF read_count=[17710277], read_ops=[49990.90], write_count=[17710276], write_ops=[49990.90], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:02:54 INF read_count=[17967899], read_ops=[52754.64], write_count=[17967899], write_ops=[52754.64], src-0, syncing rdb, size=[91 MiB/492 MiB]
2024-05-15 07:02:59 INF read_count=[18228283], read_ops=[52953.06], write_count=[18228282], write_ops=[52953.06], src-1, syncing rdb, size=[12 MiB/5.4 GiB]
2024-05-15 07:03:04 INF read_count=[18483977], read_ops=[50480.75], write_count=[18483976], write_ops=[50479.75], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:03:09 INF read_count=[18744977], read_ops=[52736.05], write_count=[18744976], write_ops=[52736.05], src-0, syncing rdb, size=[92 MiB/492 MiB]
2024-05-15 07:03:14 INF read_count=[19007607], read_ops=[52773.57], write_count=[19007607], write_ops=[52773.57], src-1, syncing rdb, size=[14 MiB/5.4 GiB]
2024-05-15 07:03:19 INF read_count=[19262889], read_ops=[50460.73], write_count=[19262888], write_ops=[50460.73], src-2, syncing rdb, size=[9.6 MiB/5.4 GiB]
2024-05-15 07:03:24 INF read_count=[19521513], read_ops=[52715.50], write_count=[19521512], write_ops=[52715.50], src-0, syncing rdb, size=[94 MiB/492 MiB]
2024-05-15 07:03:29 INF read_count=[19780249], read_ops=[52266.16], write_count=[19780248], write_ops=[52266.16], src-1, syncing rdb, size=[16 MiB/5.4 GiB]
2024-05-15 07:03:34 INF read_count=[20033352], read_ops=[49237.11], write_count=[20033351], write_ops=[49237.11], src-2, syncing rdb, size=[15 MiB/5.4 GiB]
2024-05-15 07:03:39 INF read_count=[20289052], read_ops=[52331.53], write_count=[20289051], write_ops=[52331.53], src-0, syncing rdb, size=[96 MiB/492 MiB]
2024-05-15 07:03:44 INF read_count=[20539697], read_ops=[47760.46], write_count=[20539696], write_ops=[47759.46], src-1, syncing rdb, size=[18 MiB/5.4 GiB]
2024-05-15 07:03:49 INF read_count=[20798161], read_ops=[50239.35], write_count=[20798160], write_ops=[50240.35], src-2, syncing rdb, size=[16 MiB/5.4 GiB]
2024-05-15 07:03:54 INF read_count=[21059095], read_ops=[53560.81], write_count=[21059094], write_ops=[53560.81], src-0, syncing rdb, size=[97 MiB/492 MiB]
2024-05-15 07:03:59 INF read_count=[21318595], read_ops=[52702.82], write_count=[21318594], write_ops=[52702.82], src-1, syncing rdb, size=[20 MiB/5.4 GiB]
2024-05-15 07:04:04 INF read_count=[21576099], read_ops=[52066.23], write_count=[21576098], write_ops=[52066.23], src-2, syncing rdb, size=[18 MiB/5.4 GiB]
2024-05-15 07:04:09 INF read_count=[21839243], read_ops=[53722.72], write_count=[21839242], write_ops=[53721.72], src-0, syncing rdb, size=[99 MiB/492 MiB]
2024-05-15 07:04:14 INF read_count=[22100307], read_ops=[53570.54], write_count=[22100307], write_ops=[53570.54], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:04:19 INF read_count=[22356541], read_ops=[50516.51], write_count=[22356540], write_ops=[50516.51], src-2, syncing rdb, size=[20 MiB/5.4 GiB]
2024-05-15 07:04:24 INF read_count=[22614206], read_ops=[51587.97], write_count=[22614205], write_ops=[51587.97], src-0, syncing rdb, size=[101 MiB/492 MiB]
2024-05-15 07:04:29 INF read_count=[22869818], read_ops=[52034.68], write_count=[22869817], write_ops=[52035.68], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:04:34 INF read_count=[23127535], read_ops=[51108.46], write_count=[23127534], write_ops=[51108.46], src-2, syncing rdb, size=[23 MiB/5.4 GiB]
2024-05-15 07:04:39 INF read_count=[23388860], read_ops=[51680.77], write_count=[23388859], write_ops=[51680.77], src-0, syncing rdb, size=[102 MiB/492 MiB]
2024-05-15 07:04:44 INF read_count=[23639642], read_ops=[49563.42], write_count=[23639641], write_ops=[49563.42], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:04:49 INF read_count=[23902105], read_ops=[50821.72], write_count=[23902104], write_ops=[50821.72], src-2, syncing rdb, size=[24 MiB/5.4 GiB]
2024-05-15 07:04:54 INF read_count=[24164546], read_ops=[52444.32], write_count=[24164545], write_ops=[52444.32], src-0, syncing rdb, size=[104 MiB/492 MiB]
2024-05-15 07:04:59 INF read_count=[24428660], read_ops=[52439.94], write_count=[24428660], write_ops=[52439.94], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:04 INF read_count=[24682924], read_ops=[49252.67], write_count=[24682923], write_ops=[49251.67], src-2, syncing rdb, size=[26 MiB/5.4 GiB]
2024-05-15 07:05:09 INF read_count=[24945077], read_ops=[53259.09], write_count=[24945076], write_ops=[53259.09], src-0, syncing rdb, size=[106 MiB/492 MiB]
2024-05-15 07:05:14 INF read_count=[25201637], read_ops=[53075.37], write_count=[25201636], write_ops=[53075.37], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:19 INF read_count=[25459819], read_ops=[51775.62], write_count=[25459818], write_ops=[51775.62], src-2, syncing rdb, size=[33 MiB/5.4 GiB]
2024-05-15 07:05:24 INF read_count=[25719125], read_ops=[53105.86], write_count=[25719124], write_ops=[53105.86], src-0, syncing rdb, size=[107 MiB/492 MiB]
2024-05-15 07:05:29 INF read_count=[25981309], read_ops=[53377.39], write_count=[25981308], write_ops=[53377.39], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:34 INF read_count=[26237376], read_ops=[51017.06], write_count=[26237375], write_ops=[51017.06], src-2, syncing rdb, size=[36 MiB/5.4 GiB]
2024-05-15 07:05:39 INF read_count=[26499263], read_ops=[52493.50], write_count=[26499262], write_ops=[52493.50], src-0, syncing rdb, size=[109 MiB/492 MiB]
2024-05-15 07:05:44 INF read_count=[26749924], read_ops=[49024.38], write_count=[26749923], write_ops=[49024.38], src-1, syncing rdb, size=[21 MiB/5.4 GiB]
2024-05-15 07:05:49 INF read_count=[27010779], read_ops=[51137.40], write_count=[27010779], write_ops=[51138.40], src-2, syncing rdb, size=[38 MiB/5.4 GiB]
2024-05-15 07:05:54 INF read_count=[27268557], read_ops=[49413.08], write_count=[27268556], write_ops=[49412.08], src-0, syncing rdb, size=[110 MiB/492 MiB]
2024-05-15 07:05:59 INF read_count=[27527170], read_ops=[52292.19], write_count=[27527169], write_ops=[52292.19], src-1, syncing rdb, size=[28 MiB/5.4 GiB]
2024-05-15 07:06:04 INF read_count=[27781327], read_ops=[51743.01], write_count=[27781326], write_ops=[51743.01], src-2, syncing rdb, size=[40 MiB/5.4 GiB]
2024-05-15 07:06:09 INF read_count=[28024480], read_ops=[52892.96], write_count=[28024480], write_ops=[52892.96], src-0, syncing rdb, size=[114 MiB/492 MiB]
2024-05-15 07:06:14 INF read_count=[28280875], read_ops=[49881.98], write_count=[28280874], write_ops=[49881.98], src-1, syncing rdb, size=[30 MiB/5.4 GiB]
2024-05-15 07:06:19 INF read_count=[28532759], read_ops=[49202.59], write_count=[28532758], write_ops=[49202.59], src-2, syncing rdb, size=[41 MiB/5.4 GiB]
2024-05-15 07:06:24 INF read_count=[28790741], read_ops=[52472.80], write_count=[28790740], write_ops=[52473.80], src-0, syncing rdb, size=[115 MiB/492 MiB]
2024-05-15 07:06:29 INF read_count=[29052131], read_ops=[53442.31], write_count=[29052130], write_ops=[53442.31], src-1, syncing rdb, size=[32 MiB/5.4 GiB]
2024-05-15 07:06:34 INF read_count=[29308580], read_ops=[49337.89], write_count=[29308580], write_ops=[49336.89], src-2, syncing rdb, size=[43 MiB/5.4 GiB]
2024-05-15 07:06:39 INF read_count=[29571116], read_ops=[52468.54], write_count=[29571115], write_ops=[52468.54], src-0, syncing rdb, size=[117 MiB/492 MiB]
2024-05-15 07:06:44 INF read_count=[29825588], read_ops=[49366.67], write_count=[29825587], write_ops=[49365.67], src-1, syncing rdb, size=[34 MiB/5.4 GiB]
2024-05-15 07:06:49 INF read_count=[30084683], read_ops=[50245.69], write_count=[30084683], write_ops=[50245.69], src-2, syncing rdb, size=[45 MiB/5.4 GiB]
2024-05-15 07:06:54 INF read_count=[30341291], read_ops=[52275.17], write_count=[30341290], write_ops=[52275.17], src-0, syncing rdb, size=[118 MiB/492 MiB]
2024-05-15 07:06:59 INF read_count=[30599904], read_ops=[50970.80], write_count=[30599903], write_ops=[50970.80], src-1, syncing rdb, size=[36 MiB/5.4 GiB]
2024-05-15 07:07:04 INF read_count=[30851763], read_ops=[45750.69], write_count=[30851762], write_ops=[45750.69], src-2, syncing rdb, size=[46 MiB/5.4 GiB]
2024-05-15 07:07:09 INF read_count=[31108368], read_ops=[52767.23], write_count=[31108367], write_ops=[52767.23], src-0, syncing rdb, size=[120 MiB/492 MiB]
2024-05-15 07:07:14 INF read_count=[31371390], read_ops=[52874.37], write_count=[31371389], write_ops=[52874.37], src-1, syncing rdb, size=[38 MiB/5.4 GiB]
2024-05-15 07:07:19 INF read_count=[31630350], read_ops=[52103.91], write_count=[31630349], write_ops=[52103.91], src-2, syncing rdb, size=[48 MiB/5.4 GiB]
2024-05-15 07:07:24 INF read_count=[31892047], read_ops=[52786.45], write_count=[31892047], write_ops=[52785.45], src-0, syncing rdb, size=[122 MiB/492 MiB]
2024-05-15 07:07:29 INF read_count=[32153894], read_ops=[53995.61], write_count=[32153893], write_ops=[53995.61], src-1, syncing rdb, size=[40 MiB/5.4 GiB]
2024-05-15 07:07:34 INF read_count=[32408574], read_ops=[49068.64], write_count=[32408573], write_ops=[49068.64], src-2, syncing rdb, size=[50 MiB/5.4 GiB]
2024-05-15 07:07:39 INF read_count=[32665123], read_ops=[51562.93], write_count=[32665122], write_ops=[51562.93], src-0, syncing rdb, size=[123 MiB/492 MiB]
2024-05-15 07:07:44 INF read_count=[32914499], read_ops=[47948.70], write_count=[32914498], write_ops=[47948.70], src-1, syncing rdb, size=[42 MiB/5.4 GiB]
2024-05-15 07:07:49 INF read_count=[33166647], read_ops=[49715.57], write_count=[33166646], write_ops=[49714.57], src-2, syncing rdb, size=[51 MiB/5.4 GiB]
2024-05-15 07:07:54 INF read_count=[33421498], read_ops=[51745.07], write_count=[33421497], write_ops=[51745.07], src-0, syncing rdb, size=[125 MiB/492 MiB]
2024-05-15 07:07:59 INF read_count=[33682866], read_ops=[54258.69], write_count=[33682866], write_ops=[54258.69], src-1, syncing rdb, size=[44 MiB/5.4 GiB]
2024-05-15 07:08:04 INF read_count=[33937440], read_ops=[49178.66], write_count=[33937439], write_ops=[49178.66], src-2, syncing rdb, size=[53 MiB/5.4 GiB]
2024-05-15 07:08:09 INF read_count=[34194516], read_ops=[52136.34], write_count=[34194516], write_ops=[52137.34], src-0, syncing rdb, size=[127 MiB/492 MiB]
2024-05-15 07:08:14 INF read_count=[34460156], read_ops=[52791.41], write_count=[34460155], write_ops=[52792.41], src-1, syncing rdb, size=[46 MiB/5.4 GiB]
2024-05-15 07:08:19 INF read_count=[34718813], read_ops=[52993.18], write_count=[34718812], write_ops=[52993.18], src-2, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:08:24 INF read_count=[34981932], read_ops=[53641.50], write_count=[34981931], write_ops=[53641.50], src-0, syncing rdb, size=[128 MiB/492 MiB]
2024-05-15 07:08:29 INF read_count=[35242936], read_ops=[52580.55], write_count=[35242936], write_ops=[52580.55], src-1, syncing rdb, size=[48 MiB/5.4 GiB]
2024-05-15 07:08:34 INF read_count=[35493305], read_ops=[48687.81], write_count=[35493304], write_ops=[48688.81], src-2, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:08:39 INF read_count=[35746383], read_ops=[52673.93], write_count=[35746382], write_ops=[52672.93], src-0, syncing rdb, size=[130 MiB/492 MiB]
2024-05-15 07:08:44 INF read_count=[35997876], read_ops=[50923.84], write_count=[35997875], write_ops=[50923.84], src-1, syncing rdb, size=[49 MiB/5.4 GiB]
2024-05-15 07:08:49 INF read_count=[36251779], read_ops=[50212.45], write_count=[36251779], write_ops=[50212.45], src-2, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:08:54 INF read_count=[36508020], read_ops=[51077.50], write_count=[36508019], write_ops=[51077.50], src-0, syncing rdb, size=[132 MiB/492 MiB]
2024-05-15 07:08:59 INF read_count=[36766406], read_ops=[51586.14], write_count=[36766406], write_ops=[51585.14], src-1, syncing rdb, size=[51 MiB/5.4 GiB]
2024-05-15 07:09:04 INF read_count=[37016192], read_ops=[50118.95], write_count=[37016191], write_ops=[50119.95], src-2, syncing rdb, size=[58 MiB/5.4 GiB]
2024-05-15 07:09:09 INF read_count=[37268578], read_ops=[51260.73], write_count=[37268577], write_ops=[51260.73], src-0, syncing rdb, size=[133 MiB/492 MiB]
2024-05-15 07:09:14 INF read_count=[37531302], read_ops=[53160.47], write_count=[37531301], write_ops=[53160.47], src-1, syncing rdb, size=[53 MiB/5.4 GiB]
2024-05-15 07:09:19 INF read_count=[37783084], read_ops=[49934.90], write_count=[37783084], write_ops=[49933.90], src-2, syncing rdb, size=[65 MiB/5.4 GiB]
2024-05-15 07:09:24 INF read_count=[38042391], read_ops=[50797.03], write_count=[38042391], write_ops=[50796.03], src-0, syncing rdb, size=[135 MiB/492 MiB]
2024-05-15 07:09:29 INF read_count=[38298443], read_ops=[52036.29], write_count=[38298443], write_ops=[52036.29], src-1, syncing rdb, size=[54 MiB/5.4 GiB]
2024-05-15 07:09:34 INF read_count=[38553375], read_ops=[51201.50], write_count=[38553374], write_ops=[51201.50], src-2, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:09:39 INF read_count=[38809552], read_ops=[50724.17], write_count=[38809552], write_ops=[50724.17], src-0, syncing rdb, size=[137 MiB/492 MiB]
2024-05-15 07:09:44 INF read_count=[39057980], read_ops=[49294.70], write_count=[39057980], write_ops=[49295.70], src-1, syncing rdb, size=[56 MiB/5.4 GiB]
2024-05-15 07:09:49 INF read_count=[39314114], read_ops=[50657.34], write_count=[39314113], write_ops=[50657.34], src-2, syncing rdb, size=[69 MiB/5.4 GiB]
2024-05-15 07:09:54 INF read_count=[39569479], read_ops=[49372.68], write_count=[39569478], write_ops=[49372.68], src-0, syncing rdb, size=[138 MiB/492 MiB]
2024-05-15 07:09:59 INF read_count=[39828890], read_ops=[51207.80], write_count=[39828889], write_ops=[51208.80], src-1, syncing rdb, size=[58 MiB/5.4 GiB]
2024-05-15 07:10:04 INF read_count=[40081031], read_ops=[50801.48], write_count=[40081031], write_ops=[50800.48], src-2, syncing rdb, size=[71 MiB/5.4 GiB]
2024-05-15 07:10:09 INF read_count=[40326906], read_ops=[48983.84], write_count=[40326906], write_ops=[48983.84], src-0, syncing rdb, size=[140 MiB/492 MiB]
2024-05-15 07:10:14 INF read_count=[40583448], read_ops=[50593.68], write_count=[40583447], write_ops=[50593.68], src-1, syncing rdb, size=[60 MiB/5.4 GiB]
2024-05-15 07:10:19 INF read_count=[40839249], read_ops=[47854.03], write_count=[40839248], write_ops=[47855.03], src-2, syncing rdb, size=[73 MiB/5.4 GiB]
2024-05-15 07:10:24 INF read_count=[41097391], read_ops=[52315.08], write_count=[41097390], write_ops=[52315.08], src-0, syncing rdb, size=[142 MiB/492 MiB]
2024-05-15 07:10:29 INF read_count=[41357067], read_ops=[52106.04], write_count=[41357066], write_ops=[52107.04], src-1, syncing rdb, size=[62 MiB/5.4 GiB]
2024-05-15 07:10:34 INF read_count=[41609520], read_ops=[47325.09], write_count=[41609519], write_ops=[47324.09], src-2, syncing rdb, size=[75 MiB/5.4 GiB]
2024-05-15 07:10:39 INF read_count=[41858437], read_ops=[50560.57], write_count=[41858436], write_ops=[50560.57], src-0, syncing rdb, size=[143 MiB/492 MiB]
2024-05-15 07:10:44 INF read_count=[42109121], read_ops=[48800.77], write_count=[42109120], write_ops=[48801.77], src-1, syncing rdb, size=[64 MiB/5.4 GiB]
2024-05-15 07:10:49 INF read_count=[42362012], read_ops=[48200.20], write_count=[42362011], write_ops=[48200.20], src-2, syncing rdb, size=[77 MiB/5.4 GiB]
2024-05-15 07:10:54 INF read_count=[42615028], read_ops=[51671.85], write_count=[42615027], write_ops=[51670.85], src-0, syncing rdb, size=[145 MiB/492 MiB]
2024-05-15 07:10:59 INF read_count=[42870740], read_ops=[51850.35], write_count=[42870739], write_ops=[51850.35], src-1, syncing rdb, size=[66 MiB/5.4 GiB]
2024-05-15 07:11:04 INF read_count=[43126378], read_ops=[50690.87], write_count=[43126377], write_ops=[50690.87], src-2, syncing rdb, size=[79 MiB/5.4 GiB]
2024-05-15 07:11:09 INF read_count=[43387713], read_ops=[52168.15], write_count=[43387712], write_ops=[52168.15], src-0, syncing rdb, size=[146 MiB/492 MiB]
2024-05-15 07:11:14 INF read_count=[43645924], read_ops=[51450.49], write_count=[43645923], write_ops=[51450.49], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:11:19 INF read_count=[43904825], read_ops=[52454.65], write_count=[43904824], write_ops=[52454.65], src-2, syncing rdb, size=[80 MiB/5.4 GiB]
2024-05-15 07:11:24 INF read_count=[44153751], read_ops=[50459.40], write_count=[44153750], write_ops=[50459.40], src-0, syncing rdb, size=[148 MiB/492 MiB]
2024-05-15 07:11:29 INF read_count=[44421938], read_ops=[54786.57], write_count=[44421937], write_ops=[54785.57], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:11:34 INF read_count=[44685727], read_ops=[52085.57], write_count=[44685726], write_ops=[52085.57], src-2, syncing rdb, size=[82 MiB/5.4 GiB]
2024-05-15 07:11:39 INF read_count=[44948625], read_ops=[52032.06], write_count=[44948624], write_ops=[52031.06], src-0, syncing rdb, size=[149 MiB/492 MiB]
2024-05-15 07:11:44 INF read_count=[45203314], read_ops=[51795.69], write_count=[45203314], write_ops=[51794.69], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:11:49 INF read_count=[45469362], read_ops=[53209.14], write_count=[45469361], write_ops=[53210.14], src-2, syncing rdb, size=[84 MiB/5.4 GiB]
2024-05-15 07:11:54 INF read_count=[45736162], read_ops=[52843.07], write_count=[45736161], write_ops=[52843.07], src-0, syncing rdb, size=[151 MiB/492 MiB]
2024-05-15 07:11:59 INF read_count=[46004678], read_ops=[54048.87], write_count=[46004677], write_ops=[54048.87], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:04 INF read_count=[46265822], read_ops=[51151.59], write_count=[46265821], write_ops=[51151.59], src-2, syncing rdb, size=[86 MiB/5.4 GiB]
2024-05-15 07:12:09 INF read_count=[46533600], read_ops=[54016.67], write_count=[46533599], write_ops=[54017.67], src-0, syncing rdb, size=[153 MiB/492 MiB]
2024-05-15 07:12:14 INF read_count=[46797499], read_ops=[53226.84], write_count=[46797498], write_ops=[53225.84], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:19 INF read_count=[47055220], read_ops=[50802.31], write_count=[47055219], write_ops=[50803.31], src-2, syncing rdb, size=[89 MiB/5.4 GiB]
2024-05-15 07:12:24 INF read_count=[47322553], read_ops=[54955.67], write_count=[47322552], write_ops=[54956.67], src-0, syncing rdb, size=[154 MiB/492 MiB]
2024-05-15 07:12:29 INF read_count=[47590010], read_ops=[54010.15], write_count=[47590009], write_ops=[54010.15], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:34 INF read_count=[47852892], read_ops=[52383.05], write_count=[47852891], write_ops=[52383.05], src-2, syncing rdb, size=[91 MiB/5.4 GiB]
2024-05-15 07:12:39 INF read_count=[48119623], read_ops=[53094.47], write_count=[48119622], write_ops=[53094.47], src-0, syncing rdb, size=[156 MiB/492 MiB]
2024-05-15 07:12:44 INF read_count=[48377279], read_ops=[53721.96], write_count=[48377278], write_ops=[53721.96], src-1, syncing rdb, size=[67 MiB/5.4 GiB]
2024-05-15 07:12:49 INF read_count=[48637262], read_ops=[49875.25], write_count=[48637261], write_ops=[49874.25], src-2, syncing rdb, size=[92 MiB/5.4 GiB]
2024-05-15 07:12:54 INF read_count=[48902793], read_ops=[53527.54], write_count=[48902792], write_ops=[53528.54], src-0, syncing rdb, size=[158 MiB/492 MiB]
2024-05-15 07:12:59 INF read_count=[49168303], read_ops=[53397.66], write_count=[49168302], write_ops=[53397.66], src-1, syncing rdb, size=[75 MiB/5.4 GiB]
2024-05-15 07:13:04 INF read_count=[49432497], read_ops=[50506.17], write_count=[49432496], write_ops=[50506.17], src-2, syncing rdb, size=[93 MiB/5.4 GiB]
2024-05-15 07:13:09 INF read_count=[49696189], read_ops=[54002.72], write_count=[49696188], write_ops=[54002.72], src-0, syncing rdb, size=[160 MiB/492 MiB]
2024-05-15 07:13:14 INF read_count=[49962493], read_ops=[54349.86], write_count=[49962493], write_ops=[54349.86], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:13:19 INF read_count=[50224960], read_ops=[50650.85], write_count=[50224959], write_ops=[50650.85], src-2, syncing rdb, size=[96 MiB/5.4 GiB]
2024-05-15 07:13:24 INF read_count=[50491048], read_ops=[53372.25], write_count=[50491047], write_ops=[53372.25], src-0, syncing rdb, size=[161 MiB/492 MiB]
2024-05-15 07:13:29 INF read_count=[50759051], read_ops=[53960.81], write_count=[50759051], write_ops=[53959.81], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:13:34 INF read_count=[51023338], read_ops=[51610.73], write_count=[51023337], write_ops=[51610.73], src-2, syncing rdb, size=[98 MiB/5.4 GiB]
2024-05-15 07:13:39 INF read_count=[51290582], read_ops=[54275.55], write_count=[51290581], write_ops=[54274.55], src-0, syncing rdb, size=[163 MiB/492 MiB]
2024-05-15 07:13:44 INF read_count=[51543883], read_ops=[49627.03], write_count=[51543882], write_ops=[49628.03], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:13:49 INF read_count=[51805970], read_ops=[51732.96], write_count=[51805970], write_ops=[51732.96], src-2, syncing rdb, size=[100 MiB/5.4 GiB]
2024-05-15 07:13:54 INF read_count=[52068968], read_ops=[51080.60], write_count=[52068967], write_ops=[51080.60], src-0, syncing rdb, size=[165 MiB/492 MiB]
2024-05-15 07:13:59 INF read_count=[52335847], read_ops=[54490.77], write_count=[52335846], write_ops=[54490.77], src-1, syncing rdb, size=[78 MiB/5.4 GiB]
2024-05-15 07:14:04 INF read_count=[52593364], read_ops=[49469.58], write_count=[52593364], write_ops=[49470.58], src-2, syncing rdb, size=[102 MiB/5.4 GiB]

其他信息（Additional Information）

sync_reader 与 redis_writer 的配置如下，其他配置项采用默认参数

[sync_reader]
cluster = true
address = "redis-master.prod.svc:6379"
username = ""
password = "xxxxxx"
tls = false
sync_rdb = true
sync_aof = true
prefer_replica = true

[redis_writer]
cluster = true
address = "xxx.usw1.cache.amazonaws.com:6379"
username = ""
password = "xxx"
tls = true
off_reply = false
suxb201 commented 6 months ago

目前版本确实性能存在问题：https://github.com/tair-opensource/RedisShake/pull/738 对于集群实例，建议开为每个分片单独启动 redis-shake 来提高性能。

cataglyphis commented 6 months ago

目前版本确实性能存在问题：#738 对于集群实例，建议开为每个分片单独启动 redis-shake 来提高性能。

@suxb201 相当于起多个 redis shake 去分别同步集群模式的各个分片，比如 3 主 3 从，就起 3 个 redis shake，每个 redis shake 只同步一个从分片，并把所同步的分片认为是 standalone 模式？

suxb201 commented 6 months ago

@cataglyphis 是的，可以这样设置。