Request to present Strimzi to TOC

[erinboyd]

Strimzi enables users to run Apache Kafka on Kubernetes. It is using the proven operator concept and aims to address the whole Apache Kafka life cycle:

Creating clusters Managing running clusters Optimizing running clusters for performance and efficiency Scaling clusters up and down Managing topics, users and permissions

Apache Kafka has emerged as a leading platform for building real-time data pipelines. It is distributed and horizontally scalable. That makes Apache Kafka a good candidate for running on top of Kubernetes. Apache Kafka is used as one of the basic building blocks for event driven and microservice based architectures. It provides a powerful tool for connecting scalable and decoupled components.

The Strimzi team believes that the project aligns closely with the Cloud Native Computing Foundation (CNCF) mission as described in section 1 of the CNCF Charter.

In particular: Apache Kafka is a complex application consisting of multiple separate components (Apache Zookeeper, Apache Kafka brokers, etc.). Strimzi packages Apache Kafka into containers and makes deploying Apache Kafka clusters as easy as if it was just a simple deployment. Strimzi provides a central process for managing Apache Kafka clusters. It aims to utilize Kubernetes features such as scalability to achieve efficient Apache Kafka deployments which run wherever Kubernetes runs. While Apache Kafka itself is not part of this proposal, we believe it also aligns closely with the CNCF mission. Strimzi can benefit applications that run on top of other CNCF projects such as Kubernetes. Being able to easily operate Apache Kafka on Kubernetes could act as an “enabler” for developers to adopt cloud native principles.

We hope that by bringing the Strimzi project under the CNCF umbrella as a neutral host, we will be able to further expand the community and the adoption of the project.

Website: http://strimzi.io/ GitHub: https://github.com/strimzi Video Reference: Strimzi Operator with Tom Bentley (Red Hat) - Operator Framework SIG Meeting July 2018 https://youtu.be/37DDC-Cy2ZI

License: Apache License, Version 2.0

[erinboyd]

@caniszczyk thanks in advance for your work to schedule this review in

[caniszczyk]

@erinboyd thanks, we will put this on the list for the @cncf/toc to consider for presentation next week

A quick question: why wouldn't this just be upstreamed to the Apache Kafka project?

[erinboyd]

@caniszczyk because it's more an enabler for Kafka to run on kube and leverage kube fundamentals that make it more consumable in a cloud native environment. Having said that, this is still a valid question we should talk in more depth with the TOC. Are you considering this for the Aug 7th presentation? Just want to give that team a heads up if so. Thanks!

[bgrant0607]

I second the question of why this doesn't belong in the Kafka project.

[tombentley]

The Apache Kafka project is extremely focused on the core Kafka broker, and Java clients. There are many examples of other components which, it could be argued, belong inside the Apache project, including:

• Other language clients
• Management tooling
• Protocol bridges and proxies

But these are all managed outside of the project. (For a fuller list, see here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem). Indeed, Kafka used to include non-Java language clients, but these were removed from the core project because developing them in parallel took effort away from, and slowed down development of, the rest of the project. Similarly, efforts to add other components to the Apache project have, in the past, been resisted. The same arguments seen in those discussions would apply equally well to Strimzi.

Developing Strimzi doesn’t require a detailed knowledge of Kafka’s internals, such as the Kafka protocol, concurrency within the brokers etc. A conceptual level of knowledge is sufficient. Strimzi relies much more on knowledge of the Kubernetes platform and how to best leverage the components it provides to run Kafka in the most efficient configuration possible. In short, it would be harder for Strimzi to attract developers with the requisite Kubernetes knowledge if it were inside the Apache Kafka project than if it was part of CNCF.

Putting these factors together, it is clear to the Strimzi contributors that CNCF is a more fitting place for Strimzi than the Apache Kafka project.

[seglo]

As a contributor to Strimzi I would like to add my support for this project to be included as a CNCF project. Strimzi is a robust solution to operate Kafka on Kubernetes. Kafka’s a critical messaging component in many streaming and microservice platforms that run on Kubernetes, but it can be a challenge to operate. As @tombentley has already pointed out, much of the Kafka ecosystem lives outside of the apache/kafka repository so that the Kafka team can constrain their attention to effectively manage Kafka server, Kafka Streams, and other client-centric, but platform agnostic implementations.

I mainly work within the Kafka ecosystem as a representative of the Fast Data team at Lightbend. There are several Kafka on Kubernetes projects in the community, but based on my research Strimzi is the most mature, well tested, and active open source, operator-based project available. It has a high degree of quality due to the dedication of a talented team of Red Hat engineers putting many hours into it for nearly a year. I would imagine that its closest equivalent project would be the Confluent Operator from Confluent itself, but this is currently a commercial and closed source offering which makes a comparison difficult.

I think that Strimzi, Apache Kafka, and the Kubernetes ecosystem in general would gain a lot from having this project included where it can evolve quickly along with Kubernetes itself. It would be great to see this endorsed as a CNCF project so that it can attract more contributors who come from a SRE or operations background.

[chris-vest]

I have contributed some documentation to the Strimzi project, and am also using Strimzi for a project at my current employer. I chose Strimzi after evaluating the other projects attempting to offer a similar experience to Strimzi, however none came close to offering what Strimzi does! If Strimzi is already so mature at such an early stage, I can't imagine how quickly it would develop if it were to be accepted as a CNCF project. Indeed, the Strimzi community is already well established, albeit small - under CNCF, Strimzi could truly flourish from the visibility CNCF exposure could provide.

[gwenshap]

As a PMC member in Apache Kafka, I'd like to mention that this contribution was not proposed to the Apache Kafka community.

While I agree with the sentiment that Apache Kafka community is focused on the core broker and rarely accepts other components, I think it is worthwhile raising it for discussion within the community rather than jumping to conclusions. Our last survey of the Kafka community shows that ~ 30% deploy Kafka on Kubernetes, to me this suggests that a Kafka operator is central enough to be considered part of the core project.

In general, I believe OSS projects belong with the community that can help maintain and grow them. So the question is - is there more relevant interest and expertise in Apache Kafka or in CNCF? IMO, it is difficult to know if you don't ask the community.

[Tombar]

As an early user of strimzi myself I would like to vouch for the project to be accepted & incorporated as a CNCF project. In its short lifespan, Strimzi operator already covers 80% of my uses cases out of the box & is already production ready, plus their community, support & roadmap are awesome.

Been Kafka a building block of major cloud-native architectures and the trend of major cloud providers to support it as a service (Azure, Amazon, etc) I believe CNCF users would benefit from a vendor-neutral open source kafka operator.

[caniszczyk]

strimzi presented on 4/9/2019: https://docs.google.com/presentation/d/1bijEpuwaaa6jR1D5PAjyW731-j6Xc1TFHJuUh_FwwK8/edit#slide=id.g25ca91f87f_0_0

If they find enough TOC sponsors, they can submit a PR with an official project proposal

[bgrant0607]

I think we should always ask such projects to send proposals to their corresponding upstream projects first.

Did anyone follow up with the PMC, as suggested here? https://github.com/cncf/toc/issues/138#issuecomment-441870920