Replication vs ReplicasPerPartition

[mayankshriv]

The question on what is the difference between replication vs replicasPerPartition frequently comes up. Opening this issue to track:

• Could these be unified to a single config replication that applies to both OFFLINE as well as REALTIME tables? We can still honor replicasPerPartition for backward compatibility.
• If there is we need both, could we document the reasoning behind it?

[Jackie-Jiang]

I think we have both because of the HLC to LLC migration, where they can potentially have different values for different consumer. Since we don't use HLC now, we should consider unifying them into one. cc @mcvsubbu

[mcvsubbu]

Here is the history: When we had only HLC, we had replication for both OFFLINE as well as REALTIME tables. In case of REALTIME, it was used to derive the number of realtime servers to consume the topic and also whether the topic was divided amongst the servers (something that only Uber used, I believe -- to split events on same topic across servers).

Therefore, this config could not be used for LLC, and we picked a new config -- replicasPerPartition. Note that replication is still used for OFFLINE tables.

I am not sure if we can declare HLC dead as yet. We can certainly poll the community and check, and if everyone agrees, we can remove the code, and proceed to treat replication as the number of replicas -- whether offline or realtime.

Another way to do this would be to use replication as the number of replicas if a table has only LLC (but then we have to make sure that they never ever add HLC to it -- having both HLC and LLC is one way of migrating from HLC to LLC). Fall back to replicasPerPartition if that replicas is not defined. This may be a more palatable way in case people are using HLC.

Happy to discuss over a small doc rather than this issue. We can then float the doc to the public as well

[Jackie-Jiang]

@mcvsubbu Thanks for sharing the history. I think it is always safe for LLC table to first check replicasPerPartition, and if it is not defined, fall back to replication.

[mcvsubbu]

Are we declaring HLC dead? I am fine with that, except if some stream comes along that cannot support LLC (I think we had a bit of a hard time with gps right?)

[Jackie-Jiang]

Are we declaring HLC dead? I am fine with that, except if some stream comes along that cannot support LLC (I think we had a bit of a hard time with gps right?)

No, we didn't yet, and that's why I was suggesting still first checking replicasPerPartition for LLC, and fall back to replication if it is not available. Long term wise, we don't want to use HLC because there are lots of known issues around it, and it is not as reliable as LLC.

[mcvsubbu]

My only problem with this approach is if someone out there wants to migrate from HLC to LLC and does not set replicasPerPartition.

Here is my suggestion: We remove any note we have regarding migration from HLC to LLC, and document that the only way you can migrate is to spin up another LLC cluster and shut down the HLC cluster. We may need to make some suggestions for moving historical data. AFAIK only LinkedIn attempted an in-place migration. It is probably not feasible anymore, even at LinkedIn with the increased ingestion and query volumes. We can then remove all support for HLC+LLC, and then go with the approach you are suggesting. Also, I don't know of anyone using HLC, so it is safe to assume that any migration may only happen in a future new stream type that may not support LLC to start with (but may eventually do so).

This will be a clean migration to deprecating replicasPerPartition in favor of replication.

@yupeng9 can help check whether Uber is still using this code, but that is not important to remove.

[mcvsubbu]

@yupeng9 can u please let us know if you are still using this setting?

[Jackie-Jiang]

cc @snleee

[yupeng9]

Missed this thread. Uber uses LLC only.