CMAK should stop using Zookeeper as Kafka connection endpoint

akamensky commented 4 years ago

Kafka is phasing out Zookeeper. All actions in Kafka are done via Kafka API and don't require direct connection to Zookeeper.

However CMAK still relies on defining Zookeeper as connection endpoint.

In our case Zookeeper is locked down (on network ACLs level), there is no direct connection to that from CMAK or anywhere else. Only from Kafka brokers.

patelh commented 4 years ago

Yep, we'll phase it out but we'll need some place to store metadata. Either directly in Kafka or RDBMS. Still to be designed.

akamensky commented 4 years ago

I don’t mean that CMAK shouldn’t use Zookeeper to store it’s own metadata. But it should not require zookeeper info to connect to Kafka cluster.

Currently to add cluster it requires zookeeper connection string. While Kafka has already deprecated using zookeeper in most of their tools and clients. The cluster discovery and Kafka communication API shouldn’t involve Zookeeper.

Where CMAK stores it’s own metadata has no connection to this whatsoever. In our case we are running CMAK with its own zookeeper.

akamensky commented 4 years ago

As a matter of fact as a user of CMAK I would highly prefer if it didn’t store any of its own data in Kafka. Because CMAK is a supporting tool. It shouldn’t pollute data in Kafka with stuff that’s not relevant there.

srikalam commented 4 years ago

In our case Zookeeper is locked down (on network ACLs level), there is no direct connection to that from CMAK or anywhere else. Only from Kafka brokers. @akamensky How are you able to run CMAK with the ACL restriction to Zookeeper since the zookeeper is the connection endpoint for the cluster in CMAK?

akamensky commented 4 years ago

@srikalam well, that is the point. We can’t after setting up the restrictions. Information is stale in CMAK and doesn’t get updated.

srikalam commented 4 years ago

@akamensky Thanks for the confirmation. We are also facing the same issue on our usecase. Hopefully CMAK get updated soon to configure Kafka Cluster using brokers instead of zk hosts

azhurbilo commented 4 years ago

The same for us: for security reason kafka zookeeper isolated from other services. It will be great to have ability to use Kafka api than direct zookeeper connection

cubeprogramming commented 3 years ago

@akamensky We are having the same issue and we agree with you that Kafka shouldn't be polluted with CMAK metadata either. I think en internal KV store should be introduced for metadata and all connection should be directed toward broker.

webodmin commented 1 year ago

We run kafka version 3.2.3 without zookeeper (using kraft). Unfortunately, it seems that we can't use CMAK (which we consider very useful and convenient) anymore due to absence of zookeeper. Please, add support for kraft in newer kafka versions.

Ashish123gs commented 1 year ago

I think we should discuss this further and agree on a approach, CMAK is a very powerful tool and is widely used.

We can still use a ZK to store CMAK configuration and only change connectivity to Kafka cluster for cluster specific information. This will also reduce development work in the initial phase as think to have a different storing place for CMAK configuration.

Using and managing a standalone ZK should be doable and many organisations do it for other use cases including Kafka.

Thoughts ?

Ashish123gs commented 1 year ago

Actually we can store CMAK configuration in a dedicated Kafka topic ( compacted ? )

ceeeekay commented 1 year ago

+1 for the option to store CMAK metadata in a Kafka topic. Now that KRaft is production-ready (as of 3.3.1), having to continue running ZK just for CMAK to be able to manage my clusters seems like a step backwards.

iamalryz commented 1 year ago

Currently to add cluster it requires zookeeper connection string. While Kafka has already deprecated using zookeeper in most of their tools and clients. The cluster discovery and Kafka communication API shouldn’t involve Zookeeper.

+1 for this feature