search

cult-of-coders / redis-oplog

Redis Oplog implementation to fully replace MongoDB Oplog in Meteor
MIT License
376 stars 81 forks source link

Question - Using Two Redis Servers For Multi-Region Support? #353

Closed evolross closed 4 years ago

evolross commented 4 years ago

I have a US-based app on Galaxy on AWS us-east-1 that uses redis-oplog to replace oplog between a lot of containers. All Galaxy, Mongo Atlas, and (one) Redis containers are all on us-east-1 in their various middleware services. Works awesome.

I now want to deploy a European version of the app on the eu-west-1 flavor of Galaxy. I'm able to deploy my app containers on Galaxy in eu-west-1, deploy read-only nodes of my database in Atlas on eu-west-1, and another Redis server on eu-west-1. It sounds easy enough at first thought if I update URLs appropriately. I was then going to use AWS Route 53 geolocation routing to route all Europe requests to the app to the eu-west-1 Galaxy version.

The problem is the app is collaborative and there's a very likely use-case that users in the United States might connect with users in Europe within the app and the two different Redis servers would not be real-time updating their shared data between the app containers correctly.

Is there a solution for this with redis-oplog? To somehow use two different Redis servers that work together?

If not, it seems like I would have to use one Redis server for all of them and all my European users would have to cross the pond every time they need a redis-oplog update. Which sort of defeats the purpose.

ramezrafla commented 4 years ago

@evolross I'll defer to @theodorDiaconu if there is a way with redis-oplog, but I doubt it. It listens / pushes to the local redis server and reacts to it. Can't see how you would do this, without having to duplicate redis-oplog to listen to 2 instances which would complicate things. This is no longer scalable.

What I was thinking in a similar use case is to create a redis cluster with an SSL connection between the two redis servers. This was all thought, haven't went through the mechanics, but don't see why it wouldn't work.

In fact, just like you read and write from a mongodb instance that is part of a cluster, you are doing the same thing with redis.

theodorDiaconu commented 4 years ago

This is not a problem of redis-oplog, this is a problem of scaling redis.

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Replication.Redis.Groups.html

There are many solutions out there for this. I would look at their official documentation first.