[Enhancement] Stretch Kafka cluster over multiple Kubernetes clusters

[jrivers96]

Hi,

We'd like to run strimzi multi-region in a stretch configuration because of the simplification to clients and failover benefits.

Is this in the roadmap or feasible?

[scholzj]

I assume with stretch cluster you mean stretched across different Kubernetes clusters running in the different regions? Correct?

[jrivers96]

Correct. Our data is in sequence and it is stateful to many end users. It would be nice to use all of the community adapters without having to make a custom one for multiregion HA.

Any idea how far away the repo might be? We could help out potentially.

[scholzj]

It is something we were thinking about but there is no work going on or any apecific plans at this point. It would be a lot of work since you would need to redesign everything from scratch but also the value is a bit questionable since it would still work only on closely collocated clusters only because of latency etc.

[scholzj]

As this is vaguely in our long term plans, I changed this to enhancement. But for the record, you should not expect this to happen any time soon.

[itaydj]

Hi, Just wanted to mention that we also like to run Strimzi in a streched cluster setup across multiple k8s clusters

[scholzj]

Triaged on 31.3.2022: Planned to be done in the future with StrimziPodSets & Co. Should be kept open.

[erszcz]

We would appreciate this feature a lot, too!

[jrivers96]

@scholzj Is there anything that we can look at related to the triage?

[scholzj]

There was not much discussion about it during the triage apart from saying that this is something we would want to have in the future. Just to be clear, we are triaging all the issues to clean them up. So it does not mean it will be available next month.

I talked a bit more about the importance of StrimziPodSets for this in this YouTube video: https://youtu.be/iSwrn1Gumx4 ... right now, we need to finish the PodSets first. Anyone interested in this feature can help by testing the StrimziPodSets in Strimzi 0.29, as that will be the basic building block.

It would be also great if everyone thinking about this feature had some thought about how they would expect such a stretch cluster to be linked together. There are many options and we definitely will not be able to support all of them (at least not initially for sure):

• The standard Kubernetes primitives used already for listeners today (Load balancers, NodePorts, Ingress, OpenShift Routes)
• Submariner
• Skupper
• Network Service Mesh
• Regular service mesh (e.g. using Istio federation)
• Something else?

It would be great to understand what would everyone prefer but also why. It would be also equally interesting why people might see some of these options as bad. That should allow us to better consider the pros and cons and decide about it in the future.

[jrivers96]

The standard primitives might be best to get started with easily, but I wonder about the observability required to run this at scale in production. Istio might be more interesting then, but at the cost of complexity.

I'm also thinking about the failure modes related to a stretch cluster. Is it possible to keep some topics local to a cluster and some that are global? Do you want your offset partitions globally replicated?

I haven't seen any blogs about stretch clusters....

[MR-GOYAL]

Hi, We are also want to run Strimzi Kafka in stretch cluster . Is it in progress or still in future plan.

[karolcienkosz]

Hi, From my perspective this feature wold be really useful. I am administrating several kafka clusters (VMs in GCP) which are stretched over multiple regions and it works quite good and stable if latency is relatively small(eg. europe-west1 and europe-west4). The only blocker agains migration my clusters to strimzi is lack of multiple k8s clusters support. Do we have any update about it?

[mustafaabasaran]

Hi, We are currently using on production but multi dc, stretch configuration has become mandatory for our business. Is there any news about it?

[fl0wx]

+1

[pnorth1]

Could this be accomplished via MirrorMaker2?

[scholzj]

@pnorth1 My view ... Mirror Maker mirrors data between two separate clusters. It has some advantages and disadvantages compared to stretched clusters:

• It is asynchronous so it cannot guarantee that all messages acknowledged to the producer are mirrored
• You need to have multiple Kafka clusters so it is more expensive to run
• Migrating the clients between the clusters is not completely straight forward and you need to plan with it in your architecture
• Unlike stretched cluster, Mirror Maker has only minimal limitations with regard to latency. The stretch cluster is great on paper - but it will need low latency between the Kubernetes clusters so IMHO the use-cases are fairly limited because in many cases, when you have such low latency you can stretch the Kubernetes cluster in the first place and make things even much easier.
• It is supported by Strimzi already today ;-)

So there is some overlap but there are also some differences.

[hiroarabay]

+1

[hiroarabay]

@scholzj

It is supported by Strimzi already today ;-)

how can we create the stretched cluster of multi region in strimzi ?

[scholzj]

Assuming with multi-region you mean multiple Kubernetes clusters, then you cannot. That is the point of this issue. If your Kubernetes cluster is stretched across multiple regions, you can do it in the same way as when running on any other Kubernetes cluster.

Also, please keep in mind that Kafka is latency sensitive, so while there are use cases for stretching across multiple Kubernetes clusters, it will not necessarily work over long distances because of Kafka limitations.

[hiroarabay]

@scholzj Has anyone actually created that Strimzi cluster? Do you know actual test results and documentation?

[scholzj]

@hiroarabay This is an open issue because it is not supported. So no, there is no documentation and you cannot create a Strimzi cluster stretched over multiple Kubernetes clusters.

[jcarcenegui]

Any updates on this? It would be fantastic to have it running across multiple Kubernetes clusters.

[audomsak]

Hi, I saw many reps mentioned to the latency. The questions are:

1. how low latency is required for the stretch cluster?
2. What is the maximum latency the stretch cluster can tolerate?
3. Is this latency among Kafka brokers, Zookeeper nodes, or between Kafka brokers and Zookeeper nodes?