danielealbano / cachegrand

cachegrand - a modern data ingestion, processing and serving platform built for today's hardware
BSD 3-Clause "New" or "Revised" License
975 stars 34 forks source link

Implement cluster mode #367

Open danielealbano opened 1 year ago

danielealbano commented 1 year ago

Implement a mechanism to allow multiple instances of cachegrand to work in a cluster.

The initial implementation should aim to a simple primary/secondary instance where all the operations are carried out on the primary instance and replicated to the secondary.

The replication should allow different guarantee levels configurable via the config file:

The initial sync mechanism can be implemented using the same logic implemented via the snapshotting although it's a bit easier as it's not necessary to keep track of the old values, if a value change menwhile a cluster node is added the new node can simply get the new value.

The protocol can be UDP based and be message oriented, to avoid packet loss we can have a number of ad-hoc fields in the header of the packet to identify the sequence, if a multi-packet message which segment it is, etc..