[Discuss] Choice of Replication Model for Full Cluster Replication

[ankitkala]

This issue to discuss the replication model to be used for full cluster replication. As of now, there are 3 modes of replication that will be supported for local replication between primary and replica shards:

• Logical Replication: Where we replay the translog operations on each replica.
• Segment Replication: Where replica shard simply syncs the segments from primary shard.
• Segment Replication with remote store: Where replica shard simply syncs the segments from primary shard's remote store.

Open Questions:

• [ ] Do we want to support all modes with Full Cluster Replication? or only a specific one?
• [ ] What replication model do we want support first(or as the only option)?

[ankitkala]

Logical Replication: Pros:

• We already have logical implementation for CCR.
• Logical Replication can allow for Replication across different OpenSearch versions.
• Should require less data transfer which can be a consideration for cross region use cases.

Cons:

• Pause and resume will only be supported upto 12 hours(we can explore longer if remote translogs can be supported with logical replication as well).
• Resume after 12 hours requires fetching the whole data and will be expensive(time & network usage).
• Catching up with leader after long pauses(under 12 hours) would be slow as it'll have to replay all pending operations. Also, It is expected to add an CPU/memory overhead to the follower. We can try to manage this with node level memory capping though.
• Requires additional compute resources on follower side which adds to the cost for search(Data Proximity) cases.

---

Segment Replication: Pros:

• Less overhead on the Follower cluster as it doesn't have to do replay operations.
• Easier pause/resume since segment are homogenous across leader and follower shard.

Cons:

• More network overhead on leader node(expected).
• Can't do replication across different OpenSearch major versions.
• Since we mimic the replication model used on leader, we'll be restricting customers to only use Segment Replication for local as well.

---

Segment Replication with Remote Store: Pros:

• Less overhead on the Follower cluster as it doesn't have to do replay operations.
• Easier pause/resume since segment are homogenous across leader and follower shard.
• Less overhead on the Leader cluster as replicas & Follower cluster can fetch the segments directly from Remote store.

Cons:

• Higher network overhead(expected).
• Can't do replication across different OpenSearch major versions.
• Dependency on remote store can potentially result in low adoption in opensource.

[ankitkala]

Why support only one combination?

Out of all 3 combinations, we expect the Segment Replication with remote store to perform the best(we'll need to publish the benchmark results after the POC).

Here are the pros ans cons of supporting only this combination: Pros:

• Less combinations equals reduced effort for us to build.
• Rather than building a generic solution, we can build more optimisations specific to remote store.
  • We can also explore going connection-less(i.e. for secondary, leader's remote store is the only source of truth for all data and metadata).
  • We can also explore replication at remote store level where leader's remote store is in sync with secondary's remote store. Thus secondary cluster only talks to its remote store.

Cons:

• We'll restrict the customer to use this particular replication model(local and cross cluster) which might hurt adoption.

PS: Though we build only one combination, design should still be extensible to enable other replication models in future.