kafka-firehose-nozzle

kafka-firehose-nozzle is a CloudFoundry (CF) nozzle for Apache Kafka. It consumes data from the Cloud Foundry Firehose and publishes it to one or more Apache Kafka topics.

You can set Kafka topic for each event type (e.g., by default, LogMessage events are publish to log-message topic). Events are decoded from the dropsonde protobuf format and published to Kafka encoded in the sonde-go JSON format.

kafka-firehose-nozzle is written in Go and uses the rakutentech/go-nozzle package.

It can be used together with rakutentech/cf-metrics-refinery to build a reliable pipeline that allows users to monitor their applications via a InfluxDB/Grafana installation.

Usage

Basic usage is,

$ kafka-firehose-nozzle [options]

The following are available options,

-config PATH       Path to configuraiton file
-username NAME     username to grant access token to connect firehose
-password PASS     password to grant access token to connect firehose
-worker NUM        Number of producer worker. Default is number of CPU core
-subscription ID   Subscription ID for firehose. Default is 'kafka-firehose-nozzle'
-debug             Output event to stdout instead of producing message to kafka
-log-level LEVEL   Log level. Default level is INFO (DEBUG|INFO|ERROR)

You can set password via UAA_PASSWORD environmental variable.

Configuration

You can configure it via .toml file. You can see the example and description of this configuration file in example directory.

Install

To install, you can use go get command,

$ go get github.com/rakutentech/kafka-firehose-nozzle

You can deploy this as Cloud Foundry application with go-buildpack.

Contribution

1. Fork (https://github.com/rakutentech/kafka-firehose-nozzle/fork)
2. Create a feature branch
3. Commit your changes
4. Rebase your local changes against the master branch
5. Run test suite with the make test-all command and confirm that it passes
6. Create a new Pull Request

Author

Taichi Nakashima