tair-opensource / RedisShake

RedisShake is a Redis data processing and migration tool.
https://tair-opensource.github.io/RedisShake/
MIT License
3.77k stars 693 forks source link

v4以上版本的writer是否有限流功能? #854

Open Minnull opened 2 weeks ago

Minnull commented 2 weeks ago

问题描述(Issue Description)

目标redis集群性能太差,迁移速度太快导致目标redis响应时间变长,命令写入超时,redis-shake挂掉。

环境信息(Environment)

日志信息(Logs)

截屏2024-08-27 15 54 29

其他信息(Additional Information)

`[sync_reader] cluster = false # set to true if source is a redis cluster tls = false # sync_rdb = true # set to false if you don't want to sync rdb sync_aof = true # set to false if you don't want to sync aof prefer_replica = true # set to true if you want to sync from replica node try_diskless = false # set to true if you want to sync by socket and source repl-diskless-sync=yes

[redis_writer] cluster = false # set to true if target is a redis cluster sentinel = false # set to true if target is a redis sentinel tls = false off_reply = false # ture off the server reply`

Minnull commented 2 weeks ago

@suxb201 大佬,这里新开了限流相关的问题

suxb201 commented 2 weeks ago

这两个值改小试试:

# redis-shake uses pipeline to improve sending performance.
# This item limits the maximum number of commands in a pipeline.
pipeline_count_limit = 1024

# Client query buffers accumulate new commands. They are limited to a fixed
# amount by default. This amount is normally 1gb.
target_redis_client_max_querybuf_len = 1024_000_000
Minnull commented 2 weeks ago

请问一下,target_redis_client_max_querybuf_len 这个1gb代表什么含义呢?有办法推出来对应的迁移速度是每秒多少吗?

比如限流配置的话:每次要迁移一个集群,可以提前知道目标集群最多可以接受每秒10M的数据。我把迁移工具的限流参数调的小于每秒10M即可。

suxb201 commented 2 weeks ago

你修改 pipeline_count_limit 改小试试先

Minnull commented 2 weeks ago

分别测试了3组case (1)pipeline_count_limit = 51200,启动后,短时间内很快就报错了 (2)pipeline_count_limit = 1024,运行一段时间还是可以复现上面的报错,比case 1坚持的时间久 (3)pipeline_count_limit = 1,在测试环境可以正常运行,直到完成迁移

以上测试,可以说明调整该参数,可以有效避免上面的报错。 在迁移新集群时,期望保证迁移速度和工具稳定性的话,应该如何合理的设置该值的大小呢?

suxb201 commented 2 weeks ago

@Minnull 我也不知道,没遇到过目的端性能太差的场景。