search

Yolean / kubernetes-kafka

Kafka cluster as Kubernetes StatefulSet, plain manifests and config
Apache License 2.0
1.84k stars 734 forks source link

What makes this different from the "official" kubernetes/contrib/statefulsets/kafka? #82

Open coco98 opened 6 years ago

coco98 commented 6 years ago

Could you add sections to the README about what makes this different from https://github.com/kubernetes/contrib/tree/master/statefulsets/kafka?

Filing this issue in case the maintainers already have some points in mind. I'm planning to take both for a spin and I can contribute once I have my thoughts in place too!

solsson commented 6 years ago

I will try to add such a section, "Alternatives" maybe.

There's also https://github.com/kubernetes/charts/tree/master/incubator/kafka

I hope this repo will be a community where small organizations (who can't have dedicated Kafka ops people or consultants) can come together. A folder inside some contrib/charts repo is a poor start for such a community. I recommend that after you've taken this for a spin, if you decide to go for a kafka setup like this, you fork the repo (private or public) and have your customizations there. With a the other templates you would start from a copy, but have no use of the upstream repo's changes. We've seen valuable contributions here recently, like the discussion in #13.

I'm very interested to hear your thoughts after the initial setups.

arthurk commented 6 years ago

@solsson I just wanted to say thanks for creating and maintaining this Kubernetes files, they're up-to-date (unlike the kubernetes contrib files, don't require helm and work great!

solsson commented 6 years ago

Thanks a lot @arthurk!

I think there is an important discussion to be had in this repo: on the level of automation. Kubernetes could be used to implement the kind of tooling that I imagine every large kafka ops team has. The "Operator" pattern would fit very nicely for automated Kafka cluster health. Built-in supervision like readiness and service discovery might solve things that often overlap features built in to other clustered backends.

Now when working with automation and monitoring I've re-read parts of the Kafka Book from this perspective. For example #95 https://github.com/Yolean/kubernetes-kafka/pull/95/commits/e6a7aec8bd4db6cb75617e05dd451f77f9a0cd54 is interesting as it tries to capture some of that knowledge.