Seeking Best Practices for Managing a ZooKeeper Cluster with Karmada

[tpiperatgod]

Please provide an in-depth description of the question you have:

Hi, I would like to understand the best practices for managing a ZooKeeper cluster with Karmada. My goal is to use a ZK Operator to maintain a cross-cluster ZooKeeper cluster. Based on my understanding of the tutorials provided by Karmada, I have attempted to conceptualize the propagation of the ZK Operator in the following manner:

1. Generate a ClusterPropagationPolicy for the CRD.
2. Generate a PropagationPolicy for the application.
3. Configure dependency propagation.

However, the actual scenario seems to differ from the tutorials provided by Karmada:

ZooKeeper cluster typically requires informing each replica of the address of other replicas in their configuration. This implies that my ZK Operator needs to be aware of the service addresses of ZK replicas in all clusters and provide feedback, correct? Therefore, I would like to inquire about any best practices or recommendations for such a scenario. Any guidance would be greatly appreciated.

Thank you for your assistance!

What do you think about this question?:

Environment:

• Karmada version: v1.7.1
• Kubernetes version: v1.25+
• Others:

[RainbowMango]

Hi, I would like to understand the best practices for managing a ZooKeeper cluster with Karmada. My goal is to use a ZK Operator to maintain a cross-cluster ZooKeeper cluster.

I'm trying to understand the case. I guess the ZooKeeper cluster is the cluster where installed ZooKeeper. There are multiple instances/replicas for a ZooKeeper. Now you want to spread the instances/replicas to multiple clusters. Am I right?

[tpiperatgod]

Hi, I would like to understand the best practices for managing a ZooKeeper cluster with Karmada. My goal is to use a ZK Operator to maintain a cross-cluster ZooKeeper cluster.

I'm trying to understand the case. I guess the ZooKeeper cluster is the cluster where installed ZooKeeper. There are multiple instances/replicas for a ZooKeeper. Now you want to spread the instances/replicas to multiple clusters. Am I right?

thanks! almost that, but "zk cluster" means a ZooKeeper application, specifically workloads like zk-0, zk-1, zk-2, etc. My described scenario is roughly as follows:

• K8s Cluster 1
  • zk-01
  • zk-02
• K8s Cluster 2
  • zk-03
  • zk-04
• K8s Cluster 3
  • zk-05

Now, this ZooKeeper application cluster consists of 5 replicas: zk-01 to zk-05. Each replica of zk needs to include information about all replicas in a single ZooKeeper application cluster. For example, in zk-01, it needs to contain the addresses of replicas zk-01 to zk-05.

I think deploying the ZK Operator independently in each cluster is not sufficient to achieve this goal, correct?

[liangyuanpeng]

I think deploying the ZK Operator independently in each cluster is not sufficient to achieve this goal, correct?

@tpiperatgod Currently yes, i and http://github.com/xline-kv/xline exploring karmada to manage cross-cluster stateful applications, seems like there are same issue.

[tpiperatgod]

Thanks for sharing, I'll be following these developments.

Currently, the solution I've come up with is to deploy a multi-cluster Operator for the application in the Karmada control plane, which will share the control plane's kubeconfig with Karmada. It would do the following:

1. Watch APP CRDs
2. Schedule APPs in multiple clusters
3. Generate Karmada propagation policy to fulfill CR distribution with APP's scheduling policy

dose this make sense?

[RainbowMango]

I can understand the general idea that you want to build a multi-cluster ZK Operator that can detect the location of each ZK replicas and feed the information( the other replicas' info) to them.

Schedule APPs in multiple clusters

Once the operator finds a ZK application, it propagates the ZK application to multiple clusters by generating a PropagationPolicy?

Generate Karmada propagation policy to fulfill CR distribution with APP's scheduling policy

Is this CR used to hold the information required by each ZK replica?

[tpiperatgod]

Is this CR used to hold the information required by each ZK replica?

It won't, the information is generated during the operator's reconciliation, and I think in a multi-cluster case the operator deployed in the control plane should coordinate this information.

Also, I think the customizing resource interpreter might help with this, but it would require the operator's author to write specific logic in it. 🫠

I'm going to dev it and if there is a more effective way, please teach me, thx!

[RainbowMango]

Welcome to share your demo and progress with us at the community meeting.. Feel free to add an agenda to the following meeting time.

[RainbowMango]

Is there any progress that can shared with us? Or is there anything we can do here?

[tpiperatgod]

There are no new developments at this time. But I think abstracting a generic CrossStatefulSet CRD is a much better solution, and I'll be looking forward to Lan's proposal, and we'll be working on a similar adaptation in our project in the next phase, which will hopefully provide more material for this solution.

[tpiperatgod]

Is there any progress that can shared with us? Or is there anything we can do here?

BTW, we used to use OveriddePolicy to update a cluster's multi-cluster policy to control the expected behaviour of the operator in that cluster, we now use PropagationPolicy to distribute a ConfigMap with global perspective information instead. Operators in member clusters can now take action based on the contents of the ConfigMap.