Managing Existing Cluster [Users, Topics, ACLs are preset] with kafka-topology-builder

[Kalinga]

Hallo, @purbon i was present at your recent talk in Linuxstamtisch. And excited to tryout kafka-topology-builder.

But I am bit afraid! I am afraid of damaging our existing Setup. We have a running Kafka cluster with some Users, Topics, and ACLs.

I fear if i try with a test.yml with some test data, then it may overwrite all other previous data, that is what i meant by GitOps and 'Single Point of Truth'.

Is there a way, i can retrive the Old Settings as a base.yml file and test further by adding something to it.

Please feel free to ask incase I am not clear with my intention.

Warm regards, Kalinga Ray

[purbon]

Hi @Kalinga , nice to see you over here.

As of now the best way to start with Kafka Topology Builder is by using the included docker setups available here.

Nowadays there is no functionality to export exciting clusters into topologies, that should not be hard to implement, and I think it might be a good idea to add. I appreciate the idea.

For your questions:

• if you pass the option --allowDelete false, no destructive operation will happen in the cluster.
• currently the destructive operations for topics works based on the provided descriptors, so in case of topics are created "for any reason" outside, it will not delete them.
• for the acls, aka user permissions, the acls are managed based on the state generated from the descriptors.

But as of anything, please don't try anything new out in production unless you're really, really brave 💪 .

I hope this helps you out.

[Kalinga]

Thanks @purbon for your Reply.

https://github.com/purbon/kafka-topology-builder/tree/master/docker has several docker-compose.yml. The idea is it is upto the enduser to create the Docker Image according to his own requirement. I have seen a Docker Image hosted at https://hub.docker.com/r/purbon/kafka-topology-builder, what is its Purpose?

What if someone needs security protocol=SASL_SSL?

By Looking at https://github.com/purbon/kafka-topology-builder/blob/master/docker/sasl/docker-compose.yml, it gives me a Feeling as if, using the Docker, an entire KafkaCluster is being Setup. But incase i have an already existing cluster, and i am interested in only Managing Topics and ACLs, then how to go About it.

I would be interested in working on 'export exciting clusters into topologies' feature, or any other interesting stuff once i really get a good hold of this Project's Basics.

Warm regards Kalinga

[purbon]

Hi @Kalinga the docker-compose setups available https://github.com/purbon/kafka-topology-builder/tree/master/docker are there to help with test and as playground, they actually help me during development or when showing people how this tool works. They can help you if you want to experiment with a fresh cluster without worrying.

The docker image available https://hub.docker.com/repository/docker/purbon/kafka-topology-builder is actually the topology builder wrapped there, so you can for example run it with docker run or inside other systems like k8s.

For your question on SASL_SSL, the docker-compose setups are pure examples, the tool works with admin client, so it will work with any configuration that kafka can use, just add your client config inside your topology builder property file.

Please let me know how the implementation of the export goes, if you want you can open a work in progress pull request and I can assist and help as much as possible.

Looking forward to that, thanks a lot for your interest,

-- Pere

[Kalinga]

Thanks @purbon for your reply! I get it, with the docker Image on https://hub.docker.com, with necesarry propertise file and descriptor.yaml ACLs can be managed. Soon i shall make a feature branch. After a short Research, i decided to use https://github.com/FasterXML/jackson-dataformats-text to serialize the ACLs info after fetching it using admin Client and serializing it to a .yaml file using FasterXML/jackson-dataformats-text' yaml backend. Give a short Feedback on my understanding.

Warm regards Kalinga

[purbon]

The project is already using jackson as a serdes, i would suggest the best way to give quick feedback on this is to have a work in progress Pull Request, there we can talk directly about code.

[purbon]

Thanks a lot for helping out with this! is much appreciate it.

[purbon]

closing this in favour of #70 as a better describing issue of the functionality we aim to fulfil here, basically an export into a set of topologies of what we currently have in a cluster. Feel free to reopen if this is not clear.