tair-opensource / RedisShake

RedisShake is a Redis data processing and migration tool.
https://tair-opensource.github.io/RedisShake/
MIT License
3.86k stars 699 forks source link

是否要求运行RedisShake节点要能够同时访问源跟目的Redis的所有节点吗? #876

Open linshenyuan00 opened 3 weeks ago

linshenyuan00 commented 3 weeks ago

问题描述(Issue Description)

两个Redis部署在不同的环境上,之间的内网IP不通,是否这种就无法同步? (即跨云环境,网络需要特别打通吗?) (我想将aws上的ElastiCache Redis OSS迁移到其他云上自建的Redis 集群上。) 目前RedisShake是运行在跟ElastiCache 同VPC下的一个云主机上面,而目的Redis集群是在另外的云的VPC上。

Please provide a brief description of the issue you encountered.

环境信息(Environment)

RedisShake 版本(RedisShake Version):v4.2.0 Redis 源端版本(Redis Source Version):7.0版本 Redis 目的端版本(Redis Destination Version):7.0版本 Redis 部署方式(standalone/cluster/sentinel):cluster

日志信息(Logs)

2024-10-29 01:52:36 INF load config from file: /home/linyh1/shake.toml 2024-10-29 01:52:36 INF log_level: [info], log_file: [/home/linyh1/data/shake.log] 2024-10-29 01:52:36 INF changed work dir. dir=[/home/linyh1/data] 2024-10-29 01:52:36 INF GOMAXPROCS defaults to the value of runtime.NumCPU [1] 2024-10-29 01:52:36 INF not set pprof port 2024-10-29 01:52:36 INF address=myredis.unwzow.clustercfg.apne1.cache.amazonaws.com:6379, reply=acffbf2015baa82698f5c41a3ca2a88173e58408 172.31.0.207:6379@1122 slave 613711e7fac6e6713090d2c911b05c9c7b02d6fc 0 1730166755000 1 connected 20623f07bbfdde339314b85f0c97029414a8a233 172.31.15.83:6379@1122 master - 0 1730166755000 2 connected 0-5461 7dcf82e533b22a7a16c11ca6d3a6d0b5169df0bd 172.31.28.8:6379@1122 slave 4dc93fd30659b0588e356ae783ea2e8bd19940e9 0 1730166756281 0 connected 7b028448b5d75d638083d55f13c712e288469936 172.31.26.195:6379@1122 slave 20623f07bbfdde339314b85f0c97029414a8a233 0 1730166755273 2 connected 4dc93fd30659b0588e356ae783ea2e8bd19940e9 172.31.46.183:6379@1122 master - 0 1730166755000 0 connected 10923-16383 613711e7fac6e6713090d2c911b05c9c7b02d6fc 172.31.18.206:6379@1122 myself,master - 0 1730166755000 1 connected 5462-10922 2024-10-29 01:52:36 INF [reader_172.31.15.83_6379] scanStandaloneReader init finished. dbs=[[0]] 2024-10-29 01:52:36 INF [reader_172.31.46.183_6379] scanStandaloneReader init finished. dbs=[[0]] 2024-10-29 01:52:36 INF [reader_172.31.18.206_6379] scanStandaloneReader init finished. dbs=[[0]] 2024-10-29 01:52:36 INF create ScanClusterReader: myredis.unwzow.clustercfg.apne1.cache.amazonaws.com:6379 2024-10-29 01:52:36 INF address=43.254.156.170:6379, reply=e325bba477417d19767fde182c649bb9d7b3ccd9 192.168.0.85:6379@16379 slave 1f2450c3dff57ab30a1114e3ea6c83d3f795573e 0 1730166756908 3 connected 0d57deaa7533fe00f37196426a7f60502f8867bd 192.168.0.83:6379@16379 master - 0 1730166754897 2 connected 5461-10922 1f2450c3dff57ab30a1114e3ea6c83d3f795573e 192.168.0.84:6379@16379 master - 0 1730166753891 3 connected 10923-16383 301dec657b976c46f4eac78e607e18d41dcd0489 192.168.0.87:6379@16379 slave 0d57deaa7533fe00f37196426a7f60502f8867bd 0 1730166755000 2 connected 8d02a021c3ff175f5d993ac9c9d3503c274de5c4 192.168.0.86:6379@16379 slave da038ed86f1a6f3ba65a3d5ce47674b986d2248c 0 1730166756000 1 connected da038ed86f1a6f3ba65a3d5ce47674b986d2248c 192.168.0.82:6379@16379 myself,master - 0 1730166754000 1 connected 0-5460 2024-10-29 01:52:37 ERR dial failed. address=[192.168.0.83:6379], tls=[false], err=[dial tcp 192.168.0.83:6379: i/o timeout] RedisShake/internal/client/redis.go:49 -> NewRedisClient() RedisShake/internal/writer/redis_standalone_writer.go:51 -> NewRedisStandaloneWriter() RedisShake/internal/writer/redis_cluster_writer.go:40 -> (*RedisClusterWriter).loadClusterNodes() RedisShake/internal/writer/redis_cluster_writer.go:23 -> NewRedisClusterWriter() RedisShake/cmd/redis-shake/main.go:101 -> main() runtime/proc.go:267 -> main() runtime/asm_amd64.s:1650 -> goexit()

其他信息(Additional Information)

.

suxb201 commented 3 weeks ago

是的,要求运行 RedisShake 的节点要能够同时访问源跟目的 Redis 的所有节点

linshenyuan00 commented 3 weeks ago

是否可以直接配置目的Redis的公网端口,来规避这种问题呢?

suxb201 commented 3 weeks ago

可以的

linshenyuan00 commented 3 weeks ago

请问应该怎么配置呢? 有具体的文档吗?

suxb201 commented 3 weeks ago

https://tair-opensource.github.io/RedisShake/

linshenyuan00 commented 3 weeks ago

我看 https://tair-opensource.github.io/RedisShake/zh/writer/redis_writer.html 只有介绍 redis_writer 如果是集群,配置一个节点就行了, 这种场景是说address 配置所有对外映射的节点吗? 还是其他参数呢?

suxb201 commented 3 weeks ago

@linshenyuan00 明白你意思了。 你是如何给目的端开公网的?如果目的端 Redis listen 的 ip 就是公网 ip,那没问题,直接配置上就能跑。 如果是通过代理开公网,你需要修改较多 RedisShake 的代码,支持手动指定目的端地址才行。

linshenyuan00 commented 3 weeks ago

好的 那我知道了 谢谢