Prometheus Exporter Plugin for OpenSearch

The Prometheus® exporter plugin for OpenSearch® exposes many OpenSearch metrics in Prometheus format.

• Preface
• Introduction
• Compatibility Matrix
• Install or Remove Plugin
• Plugin Configuration
  • Static settings
  • Dynamic settings
• Usage
• Build from Source
• Testing
• License
• Trademarks & Attributions

Preface

This repository contains two projects that are related, but each can be used independently:

1. The first project is a Java plugin – Prometheus Exporter Plugin for OpenSearch® – this code lives in the main branch.
2. The second project is a Jsonnet bundle – OpenSearch® Mixin – this code lives in the mixin branch.

Introduction

This plugin was started as a fork of Prometheus exporter for Elasticsearch® (it was forked in version 7.10.2.0; commit hash: 8dc7f85) utilizing OpenSearch plugin template. It uses the official Prometheus Java Simpleclient.

Currently, the available metrics are:

• Cluster status
• Nodes status:
  • JVM
  • Indices (global)
  • Transport
  • HTTP
  • Scripts
  • Process
  • Operating System
  • File System
  • Circuit Breaker
• Indices status
• Cluster settings (notably disk watermarks that can be updated dynamically)

Compatibility Matrix

NOTE: OpenSearch plugins much match exactly in major.minor.path version to the OpenSearch instance it's running in, e.g. plugins versions 2.8.0.x work only with OpenSearch 2.8.0, see also Opensearch Plugins. Therefore you must keep your prometheus-exporter-plugin-for-opensearch version in sync with your OpenSearch version.

(*) If you are looking for plugin releases supporting earlier (<= 1.2.4) OpenSearch versions please visit https://github.com/aparo/opensearch-prometheus-exporter/releases.

Install or Remove Plugin

You need to install the plugin on every OpenSearch node that will be scraped by Prometheus.

To install the plugin:

./bin/opensearch-plugin install https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.17.1.0/prometheus-exporter-2.17.1.0.zip

To remove the plugin.

./bin/opensearch-plugin remove prometheus-exporter

For more info about plugin CLI, visit: https://opensearch.org/docs/latest/opensearch/install/plugins/

Plugin Configuration

Static settings

Static settings are configured in config/opensearch.yml.

Metric name prefix

All metric names share common prefix, by default set to opensearch_.

The value of this prefix can be customized using setting:

prometheus.metric_name.prefix: "opensearch_"

Dynamic settings

Dynamic settings are configured in config/opensearch.yml but they can also be updated at any time via REST API.

Index level metrics

Whether to export detailed index level metrics or not. Default value: true.

Detailed index level metrics can represent a lot of data and can lead to high-cardinality labels in Prometheus. If you do not need detailed index level metrics it is recommended to disable it via setting:

prometheus.indices: false

Cluster settings

Whether to export cluster settings metrics or not. Default value: true.

To disable exporting cluster settings use:

prometheus.cluster.settings: false

Nodes filter

Metrics include statistics about individual OpenSearch nodes. By default, only statistics from the node that received the request are included.

Prometheus exporter can be configured to include statistics from other nodes as well. This filter is directly utilizing OpenSearch Node filters feature. Default value: "_local".

For example to get stats for all cluster nodes from any node use settings:

prometheus.nodes.filter: "_all"

Indices filter

Prometheus exporter can be configured to filter indices statistics from selected indices. To target all indices, use "" or * or _all or null. Default value: "".

For example to filter indices statistics:

prometheus.indices_filter.selected_indices: "log-*,*log,*log*,log*-test"

Users can select which option for filtering indices statistics. Default value: "STRICT_EXPAND_OPEN_FORBID_CLOSED".

For example to select an option:

prometheus.indices_filter.selected_option: "LENIENT_EXPAND_OPEN"

Here is a list of indices options available for selection

STRICT_EXPAND_OPEN: indices options that requires every specified index to exist, expands wildcards only to open indices and allows that no indices are resolved from wildcard expressions (not returning an error).
STRICT_EXPAND_OPEN_HIDDEN: indices options that requires every specified index to exist, expands wildcards only to open indices, includes hidden indices, and allows that no indices are resolved from wildcard expressions (not returning an error).
STRICT_EXPAND_OPEN_FORBID_CLOSED: indices options that requires every specified index to exist, expands wildcards only to open indices, allows that no indices are resolved from wildcard expressions (not returning an error) and forbids the use of closed indices by throwing an error.
STRICT_EXPAND_OPEN_HIDDEN_FORBID_CLOSED: indices options that requires every specified index to exist, expands wildcards only to open indices, includes hidden indices, and allows that no indices are resolved from wildcard expressions (not returning an error) and forbids the use of closed indices by throwing an error.
STRICT_EXPAND_OPEN_FORBID_CLOSED_IGNORE_THROTTLED: indices options that requires every specified index to exist, expands wildcards only to open indices, allows that no indices are resolved from wildcard expressions (not returning an error), forbids the use of closed indices by throwing an error and ignores indices that are throttled.
STRICT_EXPAND_OPEN_CLOSED: indices option that requires every specified index to exist, expands wildcards to both open and closed indices and allows that no indices are resolved from wildcard expressions (not returning an error).
STRICT_EXPAND_OPEN_CLOSED_HIDDEN: indices option that requires every specified index to exist, expands wildcards to both open and closed indices, includes hidden indices, and allows that no indices are resolved from wildcard expressions (not returning an error).
STRICT_SINGLE_INDEX_NO_EXPAND_FORBID_CLOSED: indices option that requires each specified index or alias to exist, doesn't expand wildcards and throws error if any of the aliases resolves to multiple indices.
LENIENT_EXPAND_OPEN: indices options that ignores unavailable indices, expands wildcards only to open indices and allows that no indices are resolved from wildcard expressions (not returning an error).
LENIENT_EXPAND_OPEN_HIDDEN: indices options that ignores unavailable indices, expands wildcards to open and hidden indices, and allows that no indices are resolved from wildcard expressions (not returning an error).
LENIENT_EXPAND_OPEN_CLOSED: indices options that ignores unavailable indices, expands wildcards to both open and closed indices and allows that no indices are resolved from wildcard expressions (not returning an error).
LENIENT_EXPAND_OPEN_CLOSED_HIDDEN: indices options that ignores unavailable indices, expands wildcards to all open and closed indices and allows that no indices are resolved from wildcard expressions (not returning an error).

Usage

Metrics are directly available at:

http(s)://<opensearch-host>:9200/_prometheus/metrics

As a sample result, you get:

# HELP opensearch_process_mem_total_virtual_bytes Memory used by ES process
# TYPE opensearch_process_mem_total_virtual_bytes gauge
opensearch_process_mem_total_virtual_bytes{cluster="develop",node="develop01",} 3.626733568E9
# HELP opensearch_indices_indexing_is_throttled_bool Is indexing throttling ?
# TYPE opensearch_indices_indexing_is_throttled_bool gauge
opensearch_indices_indexing_is_throttled_bool{cluster="develop",node="develop01",} 0.0
# HELP opensearch_jvm_gc_collection_time_seconds Time spent for GC collections
# TYPE opensearch_jvm_gc_collection_time_seconds counter
opensearch_jvm_gc_collection_time_seconds{cluster="develop",node="develop01",gc="old",} 0.0
opensearch_jvm_gc_collection_time_seconds{cluster="develop",node="develop01",gc="young",} 0.0
# HELP opensearch_indices_requestcache_memory_size_bytes Memory used for request cache
# TYPE opensearch_indices_requestcache_memory_size_bytes gauge
opensearch_indices_requestcache_memory_size_bytes{cluster="develop",node="develop01",} 0.0
# HELP opensearch_indices_search_open_contexts_number Number of search open contexts
# TYPE opensearch_indices_search_open_contexts_number gauge
opensearch_indices_search_open_contexts_number{cluster="develop",node="develop01",} 0.0
# HELP opensearch_jvm_mem_nonheap_used_bytes Memory used apart from heap
# TYPE opensearch_jvm_mem_nonheap_used_bytes gauge
opensearch_jvm_mem_nonheap_used_bytes{cluster="develop",node="develop01",} 5.5302736E7
...

Configure the Prometheus target

On your Prometheus servers, configure a new job as usual.

For example, if you have a cluster of 3 nodes:

- job_name: opensearch
  scrape_interval: 10s
  metrics_path: "/_prometheus/metrics"
  static_configs:
  - targets:
    - node1:9200
    - node2:9200
    - node3:9200

Of course, you could use the service discovery service instead of a static config.

Just keep in mind that metrics_path must be /_prometheus/metrics, otherwise Prometheus will find no metric.

Build from Source

To build the plugin you need JDK 17:

./gradlew clean build

If you have doubts about the system requirements, please check the CI.yml file for more information.

Testing

Project contains:

• integration tests implemented using rest layer framework
• internal cluster tests

Complete test suite is run using:

./gradlew clean check

To run individual integration rest test file use:

./gradlew :yamlRestTest \
  -Dtests.method="test {yaml=/20_11_index_level_metrics_disabled/Dynamically disable index level metrics}"

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Trademarks & Attributions

Prometheus is a registered trademark of The Linux Foundation. OpenSearch is a registered trademark of Amazon Web Services. Elasticsearch is a registered trademark of Elasticsearch BV.