Replicate segments in multiple data centers

[sl1316]

Is your feature request related to a problem? Please describe. we have a use case where requires data backup across multiple data centers and needs strong consistency. The ideal view is each segment is replicated to three clusters located at three different data centers. pulsar supports using multiple clusters as a large bookie pool but does not provide the selection logic to ensure each data center has at least one replica.

Describe the solution you'd like allow the segment replicas to be placed at each cluster(or data center). this is useful for use cases like banking in case of disaster. to my understanding, we have the cluster/topic relationship stored in zookeeper so we can probably have the admin api to enter the replication clusters similar to the allowed-clusters. or we can do it as part of the bookie config file so it will take effect at pulsar instance level Describe alternatives you've considered We also looked into the geo-replication approach but it's async and increases the cost (number of replicas * cost in a single data center) Additional context Add any other context or screenshots about the feature request here.

[Jason918]

Is geo-replication what you're look for? See https://pulsar.apache.org/docs/en/administration-geo

[sl1316]

@Jason918 as I mentioned in the alternative approach, it's async and increases the cost. what we want is to have one segment replica per data center and they are replicated in a synchronous way similar to Ack Quorum Size which guarantees the data replicated synchronously to x bookie hosts before returning success back to the client. basically, it's something like "Ack one copy per cluster."

[Jason918]

as I mentioned in the alternative approach, it's async and increases the cost. what we want is to have one segment replica per data center and they are replicated in a synchronous way similar to Ack Quorum Size which guarantees the data replicated synchronously to x bookie hosts before returning success back to the client. basically, it's something like "Ack one copy per cluster."

Oh, sorry for missed alternatives part.

This is quite an interesting topic. May I ask, what is the status of your producer and consumers in these data centers. Possible situation contains:

• Produce and consume in one data center, just want remote replicas to recover data.
• Produce in one and consume in all. Some async consuming case.
• Produce in all and only consuming local data. Typical in multi-site high availability
• etc.

[sl1316]

we have use cases for all the above scenarios and the most common one is producing in one and consuming in all(3 data centers at most). AFAIK, bookies already have the mechanism to replicate segments in different hosts so I guess it's not hard to just add additional config to make sure the hosts are located in different datacenters.

[sl1316]

even for use case 1, I do see use cases where strong consistency is required for backup data such as finance department

[codelipenghui]

@sl1316 If I understand correctly, you need one pulsar cluster, but the cluster can be across multiple regions, and the data replicas should be distributed to different regions right?

[hangc0276]

@sl1316 Does the region aware placement policy can fulfill your requirements? You can configure the three replica located into three different data centers on each entry's write. https://pulsar.apache.org/docs/en/administration-isolation/

[sl1316]

@sl1316 If I understand correctly, you need one pulsar cluster, but the cluster can be across multiple regions, and the data replicas should be distributed to different regions right?

not really. there is no guarantee the replicas will sit in different regions. it could be possible that all the replicas of a segment colocated at nodes of the same region.

[sl1316]

@sl1316 Does the region aware placement policy can fulfill your requirements? You can configure the three replica located into three different data centers on each entry's write. https://pulsar.apache.org/docs/en/administration-isolation/

could you elaborate on this? it looks like only limiting the tenants/namespace to a specific set of nodes. what I want to achieve is to have the segment replicated to different regions(one replica per region)

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.