Use Strimzi.io and custom resource files to describe the Apache Kafka cluster and their topics

[matzew]

Simple, k8s native deployment for an Apache Kafka cluster, like Strimzi.

With this, provisioning Apache Kafka cluster(s) is easy, as the cluster is described via a CR file, like:

apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Similar, the Strimzi.io has a CRD for topics as well, e.g.:

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 1
  replicas: 1
  config:
    retention.ms: 7200000
    segment.bytes: 1073741824

In Strimzi there are a few k8s operators that watch these resources for changes.

Additional Info

For future interest, it's also possible to describe users via a different CRD, using a CR file like: https://github.com/strimzi/strimzi-kafka-operator/blob/master/examples/user/kafka-user.yaml

[matzew]

One task here could be an additional installation instruction for the Strimzi.io operator(s) to manage a kafka cluster.

The Kafka Bus provisioner could leverage these Strimzi CRs

[matzew]

@evankanderson I think with the Strimzi project we could have a very kube-native ;-) implementation here for provision a Kafka bus

[evankanderson]

This looks like a great opportunity for integration when we extract many of the processors with external dependencies from the core repo (probably in release 0.2). I believe the current plan is to enable and encourage additional Source and Channel implementations once we have a tested 0.1 release. At that point, we would perform the following changes:

knative/eventing	knative/eventing-contrib	other repos
in-memory	kafka	nats?
k8sevents	gcppubsub	activemq?
ingress	github	slack?
		twitter

Where I expect that knative/eventing-contrib would contain a set of additional implementations maintained by the knative authors, but otherwise identical to other repos. (We need to practice making our changes backwards compatible cross-repo, and we'd be maintaining the eventing-contrib because they were useful to us, but wouldn't consider these "privileged" over other repos -- there would be a single catalog of channels and sources which included both contrib and extended repos.)

[evankanderson]

@scothis is probably interested in Kafka topics.

[scothis]

@matzew Thanks for the reference. Strimzi looks like an interesting project. Do you have a sense of how many users will want to have Kafka running in the same cluster as Knative? Strimzi seems to provide the most benefit if they are in the same cluster.

For provisioners, we'll need to strike a balance between making it easy to get up and running with operational concerns. The current Kafka bus is configured by providing a URL, and is decoupled from how Kafka itself is provisioned. The Kafka cluster could be provisioned by Strimzi.

[matzew]

The current Kafka bus is configured by providing a URL, and is decoupled from how Kafka itself is provisioned. The Kafka cluster could be provisioned by Strimzi.

I've seen the KAFKA_BROKERS hook, and indeed my motivation was to use Strimzi as a "kube-native" Kafka provisioning model.

Another interesting point is, that when the Knative Kafka-Bus implementation creates topics, it could also be done by the Strimzi's TopicOperator watching the kind: KafkaTopic CR files. I believe the Kafka bus currently relies on the auto.create.topics.enable property set to true.

[matzew]

Strimzi seems to provide the most benefit if they are in the same cluster.

Strimzi supports three external listeners (route, loadbalancer, nodeport), that are used to connect to a Kafka cluster from outside of an Kubernetes environment. Wiring could be done via the KAFKA_BOOTSTRAP_SERVERS (or currently KAFKA_BROKERS) hook. no?

[scothis]

@matzew would you like to add a section to the Kafka Bus readme showing how to configure the existing bus to use a Strimzi provisioned Kafka? Free free to place any yaml files you may need under /config/buses/kafka/strimzi

Having the bus create Strimzi KafkaTopic resources is a much larger change. Creating a topic isn't particularly difficult and probably isn't worth requiring users to install more CRDs and controllers. It's worth keeping in mind as we migrate the current Bus model to the new Provisioner model.

[matzew]

sure, will do !

On Wed 3. Oct 2018 at 17:05, Scott Andrews notifications@github.com wrote:

@matzew https://github.com/matzew would you like to add a section to the Kafka Bus readme https://github.com/knative/eventing/blob/master/config/buses/kafka/README.md showing how to configure the existing bus to use a Strimzi provisioned Kafka? Free free to place any yaml files you may need under /config/buses/kafka/strimzi

Having the bus create Strimzi KafkaTopic resources is a much larger change. Creating a topic isn't particularly difficult and probably isn't worth requiring users to install more CRDs and controllers. It's worth keeping in mind as we migrate the current Bus model to the new Provisioner model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/knative/eventing/issues/485#issuecomment-426671854, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJnzq5NyCzO-07jvwF-xrKFmL9XGIqwks5uhNJNgaJpZM4XDz2v .

-- Sent from Gmail Mobile