= kafka-idle-topics

image:https://travis-ci.com/abraham-leal/kafka-idle-topics.svg?branch=main["Build Status", link="https://travis-ci.com/abraham-leal/kafka-idle-topics"]

Tooling to find topics that are not utilized in your Kafka cluster.

Ensure efficient usage of your topics in https://www.confluent.io/confluent-cloud/[Confluent Cloud] by discovering topics that aren't being used.

kafka-idle-topics evaluates an idle topic through these criteria:

• It has no Consumer Groups (including inactive consumer groups that have offsets committed).
• It has no active production during the time of assessment (30 seconds while running by default).
• It has no data within its partitions.

To download, go to Run for different ways of usage. It is also possible to download a release. The output will be written to a file in the same directory. The name of this file is configurable.

== Usage

[source,bash]

Usage of ./kafka-idle-topics: -allowList value A comma delimited list of topics to evaluate. It also accepts a path to a file containing a list of topics. -bootstrap-servers string Address to the target Kafka Cluster. Accepts multiple endpoints separated by a comma. Can be set using env variable KAFKA_BOOTSTRAP -disallowList value A comma delimited list of topics to exclude from evaluation. It also accepts a path to a file containing a list of topics. -filename string Custom filename for the output if needed. (default "idleTopics.txt") -hideInternalTopics Hide internal topics from assessment. -hideTopicsPrefixes value Disqualify provided prefixes from assessment. A comma delimited list. It also accepts a path to a file containing a list. -idleMinutes int Amount of minutes a topic should be idle to report it. Can be set using env variable KAFKA_IDLE_MINUTES -kafkaSecurity string Type of connection to attempt. Options: plain_tls, plain (no tls), tls (one-way), none. (default "none") -password string Password in the PLAIN module. Can be set using env variable KAFKA_PASSWORD -productionAssessmentTimeMs int Timeframe to assess active production. (default 30000) -skip string Filtering to skip. Options are: production, consumption, storage. This can be a comma-delimited list. -username string Username in the PLAIN module. Can be set using env variable KAFKA_USERNAME -version Print the current version and exit

== Security

kafka-idle-topics supports the following security set-ups:

• plain_tls: SASL_SSL w/ PLAIN
• plain: SASL_PLAINTEXT w/ PLAIN (No TLS)
• tls: SSL (One-way TLS)
• none: PLAINTEXT (Default)

To add certificates the tool can trust for TLS connections, ensure they are located in the subdirectories go seeks. Commonly it is /etc/ssl/certs/.

== Run

You can run kafka-idle-topics in multiple configurations:

=== Docker

Example with Confluent Cloud:

[source,bash]

docker run \ -e KAFKA_BOOTSTRAP=<>\ -e KAFKA_USERNAME=<>\ -e KAFKA_PASSWORD=<>\ abrahamleal/kafka-idle-topics:latest

=== Locally

Downloading with Go, example with Confluent Cloud:

[source,bash]

go get github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics go install github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics $GOPATH/bin/kafka-idle-topics -bootstrap-servers <> -username <> -password <> -kafkaSecurity plain_tls

=== Build From Source

Git clone, example with Confluent Cloud:

[source,bash]

git clone https://github.com/abraham-leal/kafka-idle-topics cd kafka-idle-topics/ go build ./cmd/kafka-idle-topics/* ./kafka-idle-topics -bootstrap-servers <> -username <> -password <> -kafkaSecurity plain_tls

=== Filtering the evaluation

It is now possible to filter the topics which are evaluated. Setting -allowList or/and -disallowList flags will accept either a comma delimited string, or a file containing comma delimited entries for topic names. If specifying a file, make sure it has an extension (such as .txt). A topic specified in -disallowList and -allowList will be disallowed by default.

Additionally, you may set -hideTopicsPrefixes to hide custom prefixes from the evaluation.

== Feature Requests / Issue Reporting

This repo tracks feature requests and issues through Github Issues. If you'd like to see something fixed that was not caught by testing, or you'd like to see a new feature, please feel free to file a Github issue in this repo, I'll review and answer at best effort.

Additionally, if you'd like to contribute a fix/feature, please feel free to open a PR for review.