Add an existing kafka cluster to CMAK

[visla-xugeng]

Hi I have an existing Kafka cluster with ZooKeeper cluster in EKS in AWS. (I built this cluster by using Strimzi operator and Kafka Kind). I failed to add this existing cluster to CMAK, after I successfully installed CMAK by cmak-operator chart. Is there anything wrong with my procedure?

kubectl -n kafka get pods
NAME                                                            READY   STATUS    RESTARTS        AGE
cn-northwest-1-ops-kafka-crd-entity-operator-546c96d747-nggbz   3/3     Running   0               38s
cn-northwest-1-ops-kafka-crd-kafka-0                            1/1     Running   0               2m41s
cn-northwest-1-ops-kafka-crd-kafka-1                            1/1     Running   0               2m41s
cn-northwest-1-ops-kafka-crd-kafka-2                            1/1     Running   0               2m41s
cn-northwest-1-ops-kafka-crd-zookeeper-0                        1/1     Running   3 (3m44s ago)   5m45s
cn-northwest-1-ops-kafka-crd-zookeeper-1                        1/1     Running   4 (3m4s ago)    5m45s
cn-northwest-1-ops-kafka-crd-zookeeper-2                        1/1     Running   0               5m45s
strimzi-cluster-operator-5c8c984b7b-tcfnt                       1/1     Running   0               4h22m

From the above screenshot, I added zookeeper hosts info into the second box. cn-northwest-1-ops-kafka-crd-zookeeper-0:2181, cn-northwest-1-ops-kafka-crd-zookeeper-1:2181, cn-northwest-1-ops-kafka-crd-zookeeper-2:2181/kafka

Then, got timeout error:

[visla-xugeng]

kubectl -n cmak get pods
NAME                               READY   STATUS      RESTARTS   AGE
cmak-76cffb897f-jfvxz              2/2     Running     0          138m
cmak-reconcile-27547128--1-89kgd   0/1     Completed   0          6m34s
cmak-reconcile-27547131--1-6scp6   0/1     Completed   0          3m34s
cmak-reconcile-27547134--1-dlgrg   0/1     Completed   0          34s

log from the pod, cmak-reconcile-xxxxxxx

2022-05-17 22:54:01,299 Connecting to cmak.cmak(172.20.198.187):2181, use_ssl: False
2022-05-17 22:54:01,303 Zookeeper connection established, state: CONNECTED
2022-05-17 22:54:01,305 Closing connection to cmak.cmak:2181
2022-05-17 22:54:01,305 Zookeeper session closed, state: CLOSED

[eshepelyuk]

1. Readme for adding kafka clusters to cmak https://github.com/eshepelyuk/cmak-operator#kafka-clusters
2. are cmak and kafka/zk in the same kind ckuster, same namespaces ?
3. are used address accesible from cmak container, i.e. calling nslookup and/or telnet

[visla-xugeng]

1: I read the link you provided, not sure if I understand correctly. My situation is that I already have a Kafka cluster created by Strimzi Operator. Then I want to use CMAK to manage this cluster visually. I do not need to use CMAK to build a new cluster. Is this correct? (Not sure if CMAK can build new kafka/zk cluster?) Based on your doc, looks like that you are using CMAK to create a new Kafka/zk cluster. Can you explain a little more here?

2: CMAK and my Kafka/zk are not in the same namespace. (But they are in the same EKS cluster.) CMAK is in the namespace, cmak, my kafka/zk cluster is in: kafka

3: About "address", do you mean Cluster Zookeeper Hosts? About Cluster Zookeeper Hosts, is this EC2 hostname, where ZK pod sit?

[eshepelyuk]

1. neither cmak nor cmak-operator docs never mentioned that cmak can be used to create kafka cluster. have absolutely no idea where did you pick up that thought from.

2. check those ZK addresses from within cmak container with nslookup and telnet. the tools are not installed in ui container, so run this command after logging to ui container

   apt update && apt -y install dnsutils telnet