search

patschuh / KafkaEsque

Kafka Development Tool
GNU General Public License v3.0
78 stars 22 forks source link
java javafx kafka kafka-tools kafka-ui

readme

KafkaEsque

What's new

Starting with version 2.0.0 KafkaEsque uses JavaFX 17 instead of JavaFX 8, and provides prepackaged builds for most operating systems (Windows, macOS, Linux) that were created with jpackage using GitHub Actions (CI). Every release on GitHub can be tracked to its 'run' (one job for each OS) in the 'Gradle Build' workflow, and the source code that was used to build it. Furthermore, jlink bundles a minimal version of the JRE/JDK with the required JavaFX libraries. Therefore, it is not necessary to have Java installed at all.

Thanks to the badass-runtime plugin for Gradle, the application is still non-modular, and it was not necessary to use Java 9 modules (also known as Project Jigsaw).

Usage

Windows

On Windows you will most likely receive a smart screen warning because the app was not signed or uploaded to Microsoft to verify it. You can safely ignore this warning. The app does not even require admin rights to install it because it is being installed for the current user only.

macOS

macOS has a similar problem with signing, except that while the dmg file that contains the app, displays the correct message, and says it was signed by a non verified developer, the app inside the dmg does not. Instead, it says that the app is damaged which is incorrect. If anyone knows how to fix this, please submit a pull request.

As most macOS user probably know, apps downloaded from the Internet are "in quarantine". You can see the flag (com.apple.quarantine) yourself if you run, e.g., ls -lh@ in the console. To install the app you have to remove the flag first with, e.g., xattr -rd com.apple.quarantine kafkaesque-2.0.0.dmg.

Linux

For Linux deb & rpm packages are provided. So every distribution that uses those, can install them. If someone actually uses Kafkaesque on another Linux distribution, feel free to submit an issue, or even a PR, if you want up-to-date packages for your distribution.

JAR

You can still run just the JAR files as before version 2.0.0, but now they require Java 17 instead of Java 8.

:information_source: This is a JavaFX application and therefore requires OpenJFX.

Alternatively you can start the JAR file with the provided helper script. Either with ./bin/KafkaEsque on macOS or Linux, or with .bin/KafkaEsque.bat on Windows.

Features

Create, remove and describe Topics

Create Topic Screenshot

Describe Topic Screenshot

Publish Messages

Publish Message Screenshot

Consume Messages

Trace Messages

By Key

Consumes Messages only keeping Messages in the result list where the message key matches the given key

in value

Consumes Messages only keeping Messages in the result list where a sequence in the message value matches the given regex

Export Messages

Mesages displayed in the message list can be exportet in csv format and played into any cluster via a message book with minimal modifications.

Topic Templates

Allows for defining and configurating topics once and apply them to different clusters with one file, see the Wiki for Details

Message Books

Allows for playing a set of Messages over different topics into a cluster, see the Wiki for details

Authentication

Within the cluster.json file it is possible to configure Authentication for Kafka and Confluent Schema Registry: Note: the secrets have to be given in the json file in plain text. This might be a security issue. Feel free to apply a PR if you want to improve this. sslEnabled controls the SSL Authentication method

Config of the Authentication can be done either in cluster.json directly or via the UI.

Example for SSL with mTLS Authentication to the broker:
    {
        "identifier": "my-mtls-secured-cluster",
        "bootstrapServers": "broker:<portofmtlslistener>",
        "sslEnabled": true,
        "keyStoreLocation": "mykeystore.jks",
        "keyStorePassword": "mykeystorepw",
        "trustStoreLocation": "mytruststore.jks",
        "trustStorePassword": "mykeystorepw"
    }
Example for SASL_SSL Authentication

saslSecurityProtocol,saslMechanism and saslJaasConfig can be provided This can also be combined with given trust and keystore configuration

    {
        "identifier": "my-mtls-secured-cluster",
        "bootstrapServers": "broker:<portofmtlslistener>",
        "saslSecurityProtocol": "SASL_SSL",
        "saslMechanism"  : "PLAIN", 
        "saslJaasConfig" : "org.apache.kafka.common.security.plain.PlainLoginModule required serviceName=kafka username=\"MYUSER\" password=\"53CR37\";"
    }
Example with Schema Registry with HTTPS and Basic Auth

The https and 'sslEnabled' is important if you want to use truststore and/or keystore otherwise those attributes are ignored and now sslContext is provided to Schema Registry client.

You can use only Basic Auth if youy SR is only protected with basic auth, you can use Token Auth if your SR is protected with an OAUTH Token, you can use only keystore+truststore if your SR is protected with mTLS or you can use both settings in parallel. schemaRegistryBasicAuthUserInfo is deprecated since token auth is supported in addition to basic auth. There is a schemaRegistryAuthMode property with possible values NONE, BASIC or TOKEN and schemaRegistryAuthConfig property with either basic auth credentials or OAuthToken.

    {
              ....
              "schemaRegistry": "https://myschemaregistry:8081", 
deprecated-> "schemaRegistryBasicAuthUserInfo": "<BasicAuthUser>:<BasicAuthPW>",
              "schemaRegistryAuthMode": "NONE|BASIC|TOKEN",
              "schemaRegistryAuthConfig": "<BasicAuthUser>:<BasicAuthPW>|<OAuthToken>:",
              ...
              "sslEnabled": true,
              "keyStoreLocation": "mykeystore.jks",
              "keyStorePassword": "mykeystorepw",
              "trustStoreLocation": "mytruststore.jks",
              "trustStorePassword": "mykeystorepw"
    }
Using SSL without domain name

In some situation you might need to use ip address for your bootstrap server and SSL. With default config the API does a host name identification which fails in those scenarios with 

java.security.cert.CertificateException: No subject alternative names matching IP address .... found

If you select the toggle "No SSL Endpoint Identification" the kafka property "ssl.endpoint.identification.algorithm" and schema-registry property "schema-registry.ssl.endpoint.identification.algorithm" are set to an empty string so that this identification is suppressed

suppress cert path validation

In some situation you might need to suppress domain name validation for schema-registry. With default config the API does a cert path validation which fails when using an ssh tunnel.

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

If you select the toggle "suppress Cert Path Validation" an empty trustmanager is set.

Settings

Check the settings.yaml in the /.kafkaesque directory for cluster independent application settings