devshawn / kafka-gitops

πŸš€Manage Apache Kafka topics and generate ACLs through a desired state file.
https://devshawn.github.io/kafka-gitops
Apache License 2.0
317 stars 71 forks source link
acl-management acls gitops kafka kafka-acls kafka-cluster kafka-dsf kafka-gitops kafka-manager kafka-topics

kafka-gitops

Java CI Maintainability License

Manage Apache Kafka topics and ACLs through a desired state file.

Overview

Kafka GitOps is an Apache Kafka resources-as-code tool which allows you to automate the management of your Apache Kafka topics and ACLs from version controlled code. It allows you to define topics and services through the use of a desired state file, much like Terraform and other infrastructure-as-code tools.

Topics and services get defined in a YAML file. When run, kafka-gitops compares your desired state to the actual state of the cluster and generates a plan to execute against the cluster. This will make your topics and ACLs match your desired state.

This tool also generates the needed ACLs for each type of application. There is no need to manually create a bunch of ACLs for Kafka Connect, Kafka Streams, etc. By defining your services, kafka-gitops will build the necessary ACLs.

This tool supports self-hosted Kafka, managed Kafka, and Confluent Cloud clusters.

Features

Getting Started

Documentation on how to install and use this tool can be found on our documentation site.

Usage

Run kafka-gitops to view the help output.

Usage: kafka-gitops [-hvV] [--no-delete] [--skip-acls] [-f=<file>] [COMMAND]
Manage Kafka resources with a desired state file.
  -f, --file=<file>   Specify the desired state file.
  -h, --help          Display this help message.
      --no-delete     Disable the ability to delete resources.
      --skip-acls     Do not take ACLs into account during plans or applies.
  -v, --verbose       Show more detail during execution.
  -V, --version       Print the current version of this tool.
Commands:
  account   Create Confluent Cloud service accounts.
  apply     Apply changes to Kafka resources.
  plan      Generate an execution plan of changes to Kafka resources.
  validate  Validates the desired state file.

Configuration

Currently, configuring bootstrap servers and other properties is done via environment variables:

To configure properties, prefix them with KAFKA_. For example:

Additionally, we provide helpers for setting the sasl.jaas.config for clusters such as Confluent Cloud.

By setting:

The following configuration is generated:

State File

By default, kafka-gitops looks for state.yaml in the current directory. You can also use kafka-gitops -f to pass a file.

An example desired state file:

topics:
  example-topic:
    partitions: 6
    replication: 3
    configs:
      cleanup.policy: compact

services:
  example-service:
    type: application
    produces:
      - example-topic
    consumes:
      - example-topic

Contributing

Contributions are very welcome. See CONTRIBUTING.md for details.

License

Copyright (c) 2020 Shawn Seymour.

Licensed under the Apache 2.0 license.