search

Restream / reindexer

Embeddable, in-memory, document-oriented database with a high-level Query builder interface.
https://reindexer.io
Apache License 2.0
763 stars 64 forks source link

Master-Slave setup on Kubernetes #45

Closed oruchreis closed 4 years ago

oruchreis commented 5 years ago

Hi, I'm trying to setup master-slave replication on kubernetes. There are a few things that I couldn't accomplish with: 1- How can I configure the replication for all existing dbs in the master? If I set master_dsn without db name like "cproto://masterip:6534/", it doesn't replicate the newly created dbs. 2- The pods on kubernetes may be down for a while and so the master pod may be down. So how can I promote a slave to master and inform the other slaves that the master has been changed? Are there any high availability solution like redis-sentinel? 3- When a pod starts, it needs to know about the master address. So I need to query if a node is the master or a slave. How can I get that if the node is slave or master?

olegator77 commented 5 years ago

Hi

  1. The object of replication is database, not server node. For example there are possible to use 2 database in single node as master and slave. Therefore each database have personal replication config. In current version of reindexer there are no requested function. For now you can implement it in application or even script : a) Request list of available databases of master by http API curl -X GET "http://<master_ip>:9088/api/v1/db" b) If in list found new databases, then create them by http API curl -X POST "http://<slave_ip>:9088/api/v1/db" --data '{"name":"<db_name>"}' c) setup replication of new database in slave: curl -X PUT "http://<slave_ip>:9088/api/v1/db/<db_name>/namespaces/%23config/items" --data '{"type":"replication","replication":{"role":"slave","master_dsn":"cproto://<master_ip>:6534/<db_name>"}}'

  2. Rigth now there are no automated cluster control solution in reindexer. We are working on it.

  3. You can check database replication config by this API call: curl -X GET http://<server_ip>:9088/api/v1/db/<db_name>/namespaces/%23config/items?filter=type%3d%27replication%27

ilyar commented 4 years ago

@oruchreis Tell me, it turned out to configure reindexer in k8s?

oruchreis commented 4 years ago

Hi @ilyar , I couldn't understand exactly what do you mean. We've implemented a few things to work with k8s in production environment. First, we've implemented a proxy application to check if the slave namespace is available to query, and second we've implemented an operator to check reindexer nodes to promote a slave to master. We also check the db configs as @olegator77 mentioned above to perform a reliable replication. So in the end, we're using extra 2 application(proxy and operator) for performing high availabilty in k8s.