Logstash-exporter

Export metrics from Logstash to Prometheus. The project was created as rewrite of existing awesome application logstash_exporter, which was also written in Go, but it was not maintained for a long time. A lot of code was reused from the original project.

Important: V2 version of the application is currently in beta. It is recommended to use V1 version in production. It is still maintained and available under v1 branch. Make sure to check the Migration section before upgrading to V2.

Important: Because of limited workforces, this project is tested only against a single Logstash version. You can check the tested version in docker-compose.yml file. Using this exporter with other versions of Logstash may not work properly (although most of the metrics should work).

Usage

Running the app

The application can be run in two ways:

• using the binary executable
• using the Docker image

Additionally Helm chart is provided for easy deployment to Kubernetes.

Flags

The application supports the following flags:

• -config: Path to the configuration file (default: config.yml). Takes precedence over EXPORTER_CONFIG_LOCATION environment variable.
• -help: Show help message.
• -version: Show semantic version.

Binary Executable

The binary executable can be downloaded from the releases page. Linux binary is available under https://github.com/kuskoman/logstash-exporter/releases/download/v${VERSION}/logstash-exporter-linux. The binary can be run without additional arguments, as the configuration is loaded from the .env file and environment variables.

Each binary should contain a SHA256 checksum file, which can be used to verify the integrity of the binary.

VERSION="test-tag" \
OS="linux" \
wget "https://github.com/kuskoman/logstash-exporter/releases/download/${VERSION}/logstash-exporter-${OS}" && \
wget "https://github.com/kuskoman/logstash-exporter/releases/download/${VERSION}/logstash-exporter-${OS}.sha256" && \
sha256sum -c logstash-exporter-${OS}.sha256

It is recommended to use the binary executable in combination with the systemd service. The application should not require any of root privileges, so it is recommended to run it as a non-root user.

Unstable (master) version

The unstable version of the application can be downloaded from the GitHub Actions. The latest successful build can be found under the Go application CI/CD workflow (already selected in the link). To download the binary, simply go to the link location, click on the latest successful build, and download the binary from the Artifacts section on the bottom of the page.

You are able to download artifact from any workflow run, not only master branch. To do that, go to GitHub Actions without master filter, select the workflow run you want to download artifact from, and download the binary from the Artifacts section.

Docker Image

The Docker image is available under kuskoman/logstash-exporter:<tag>. You can pull the image using the following command:

docker pull kuskoman/logstash-exporter:<tag>

You can browse tags on the Docker Hub.

The Docker image can be run using the following command:

docker run -d \
    -p 9198:9198 \
    -v $(pwd)/config.yml:/app/config.yml \
    kuskoman/logstash-exporter:<tag>

Unstable (master) image

The unstable version of the Docker image can be downloaded from the GitHub Container Registry. To pull the image from command line simply use:

docker pull ghcr.io/kuskoman/logstash-exporter:master

The Docker image can be run using the following command:

docker run -d \
    -p 9198:9198 \
    -v $(pwd)/config.yml:/app/config.yml \
    ghcr.io/kuskoman/logstash-exporter:master

Helm Chart

The Helm chart has its own README.

Endpoints

• /metrics: Exposes metrics in Prometheus format.
• /health: Returns 200 if app runs properly.

Configuration

The application is now configured using a YAML file instead of environment variables. An example configuration is as follows:

Important: The servers section got renamed to instances. For now both names are supported, but the servers name will be removed in the future.

logstash:
  instances:
    - url: "http://logstash:9600" # URL to Logstash API
    - url: "http://logstash2:9600"
server:
  host: "0.0.0.0" # Host on which the application will be exposed (default: all interfaces)
  port: 9198      # Port on which the application will be exposed
logging:
  level: "debug"  # Log level (debug, info, warn, error)

All configuration variables can be checked in the config directory.

Previously the application was configured using environment variables. The old configuration is no longer supported, however a migration script is provided to migrate the old configuration to the new one. See more in the Migration section.

Migration

From v1 to v2

With Migration Script

1. Backup Configuration: Save your existing .env file.
2. Run Migration Script: Execute ./scripts/migrate_v1_to_v2.sh [path/to/.env]. If your .env is in the default location, simply run ./scripts/migrate_v1_to_v2.sh. You can use it via make: make migrate-v1-to-v2.
3. Save New Configuration: Redirect output to new_config.yaml using ./scripts/migrate_v1_to_v2.sh.sh > config.yaml.
4. Set EXPORTER_CONFIG_LOCATION: Update this environment variable to the path of config.yaml (or leave the default).
5. Test Application: Ensure the application runs correctly with the new configuration.

Without Migration Script

1. Backup Configuration: Keep a record of your current environment variables.
2. Create YAML File: Manually create a config.yaml file following the new format.
3. Transfer Settings: Copy values from your .env file or noted environment variables into the corresponding sections in new_config.yaml.
4. Set EXPORTER_CONFIG_LOCATION: Point this environment variable to new_config.yaml.
5. Test Application: Check if the application functions properly with the new configuration.

Building

Makefile

Available Commands

• make all: Builds binary executables for all OS (Win, Darwin, Linux).
• make run: Runs the Go Exporter application.
• make run-and-watch-config: Runs the Go Exporter application with watching the configuration file.
• make build-linux: Builds a binary executable for Linux.
• make build-darwin: Builds a binary executable for Darwin.
• make build-windows: Builds a binary executable for Windows.
• make build-linux-arm: Builds a binary executable for Linux ARM.
• make build-docker: Builds a Docker image for the Go Exporter application.
• make build-docker-multi: Builds a multi-arch Docker image (amd64 and arm64).
• make clean: Deletes all binary executables in the out directory.
• make test: Runs all tests.
• make test-coverage: Displays test coverage report.
• make compose: Starts a Docker-compose configuration.
• make wait-for-compose: Starts a Docker-compose configuration until it's ready.
• make compose-down: Stops a Docker-compose configuration.
• make verify-metrics: Verifies the metrics from the Go Exporter application.
• make pull: Pulls the Docker image from the registry.
• make logs: Shows logs from the Docker-compose configuration.
• make minify: Minifies the binary executables.
• make install-helm-readme: Installs readme-generator-for-helm tool.
• make helm-readme: Generates Helm chart README.md file.
• make clean-elasticsearch: Cleans Elasticsearch data. The command may take a very long time to complete.
• make clean-prometheus: Cleans Prometheus data.
• make upgrade-dependencies: Upgrades all dependencies.
• make migrate-v1-to-v2: Migrates configuration from v1 to v2.
• make update-readme-descriptions: Update Makefile descriptions in main README.md.
• make update-snapshots: Updates snapshot for test data and runs tests.
• make help: Shows info about available commands.

File Structure

The main Go Exporter application is located in the cmd/exporter/main.go file. The binary executables are saved in the out directory.

Helper Scripts

Application repository contains some helper scripts, which can be used to improve process of building, testing, and running the application. These scripts are not useful for the end user, but they can be useful for all potential contributors. The helper scripts are located in the scripts directory.

add_metrics_to_readme.sh

This script is used to add metrics table to the README.md file. Usage:

./scripts/add_metrics_to_readme.sh

create_release_notes.sh

This script is used to create release notes for the GitHub release. Used primarily by the CI workflow.

generate_helm_readme.sh

This script is used to generate Helm chart README.md file. The readme contains all the configuration variables from the values.yaml file.

install_helm_readme_generator.sh

This script is used to install readme-generator-for-helm tool. The tool is used to generate Helm chart README.md file. The script installs the tool under helm-generator directory.

verify_metrics.sh

This script is used to verify the metrics from the Go Exporter application. Can be used both locally and in the CI workflow.

./scripts/verify_metrics.sh

Testing process

The application contains both unit and integration tests. All the tests are executed in the CI workflow.

Unit Tests

Unit tests are located in the same directories as the tested files. To run all unit tests, use the following command:

make test

Integration Tests

Integration tests checks if Prometheus metrics are exposed properly. To run them you must setup development docker-compose file.

make wait-for-compose

Then you can run the tests:

make verify-metrics

Grafana Dashboard

A Grafana Dashboard designed for metrics from Logstash-exporter on Kubernetes is available at grafana.com/grafana/dashboards/18628-logstash-on-kubernetes-dashboard/. This dashboard's JSON source is at excalq/grafana-logstash-kubernetes.

(If not using Kubernetes, change $pod to $instance in the JSON.)

Additional Information

This projects code was reviewed by Boldly Go in an awesome video, which in a huge way helped me to improve the code quality.

Contributing

If you want to contribute to this project, please read the CONTRIBUTING.md file.

Metrics

Table of exported metrics: