search

fullstorydev / solr-bench

Solr benchmarking and load testing harness
Apache License 2.0
17 stars 10 forks source link

Export benchmarking data via Prometheus exporter at endpoint /metrics #84

Closed patsonluk closed 11 months ago

patsonluk commented 11 months ago

Description

Currently solr-bench export test results to a json file upon test completion. However, it would also be useful to have the test metrics be exposed via Prometheus /metrics endpoint such that data can be graphed by other tools such as Grafana

Solution

Added new config parameter prometheus-export to the workflow (top-level) config. 

{
  ...
  "prometheus-export": { "port": 1234, "type-label": "facet" }
}

port is the port used for prometheus /metrics endpoint (default to 11100, take note that we picked this to avoid the default 9100 port used for prometheus node-exporter) type-label is a custom label that user can define to better group metrics. default to unknown. There's also an override prometheus-type-label under query-benchmark in case if user wants more granular control on such label to per benchmark setup.

To enable Prometheus with defaults, simply add "prometheus-export": {}

Histogram is used to store the metrics with labels:

  1. method : the http Method used for the call
  2. path : the relative path used . For example /select, /update etc
  3. type : the custom label described above. Default to unknown

The current implementation explicitly declares the metrics to be captured (as histogram) and only capture metrics from QueryBenchmark and IndexBenchmark. Other benchmarking tasks are not prometheus enabled ATM.

Take note that the Prometheus endpoint server would only be started if prometheus-export is defined with valid metrics to export. (ie if the benchmark only does restart, it wouldn't start the prometheus export endpoint)

Remarks

This PR also adds some extra features, fixed some bugs:

  1. stress.sh would now read the json config file with a new param extra-jvm-args, which the value will be added to the java startup call. This is useful for SSL support
  2. Fixed an issue with dangling ZK client connection that prevents proper shutdown
patsonluk commented 11 months ago

@chatman Adding ability to export benchmarking metrics via grafana /metrics endpoint (different from the existing prometheusMetrics collector which collects prometheus metrics from the current host)

Also tagging @nginthfs and @hiteshk25 😊

Fixed some inconsistent whitespace vs tab and indentation in the files, please use https://github.com/fullstorydev/solr-bench/pull/84/files?w=1 for code review!

patsonluk commented 11 months ago

Changes look good! Thanks..

Is it possible to include a sample configuration exposing some of these changes? Not a blocker, we can do it later also.

Thank you so much for the review!

As for the config, I thought of adding new one or changing existing one, but it could be a bit hard for users as it's not super trivial which config would have the settings.

As a part of this PR, we did add description to the README.md and the sample usage: https://github.com/fullstorydev/solr-bench/pull/84/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R88 .