Enhanced metrics

[ujibang]

Brief overview

RESTHeart currently provides a metrics feature described here. It is embedded in the plugin mongo and only collects metrics for requests handled by the MongoService

We want:

• keep using the dropwizard library
• refactor the metrics feature in a new, dedicated plugin
• collect prometheus metrics for all services
• follow prometheus conventions for metrics names and labels
• collect metrics for http requests and the jvm (using dropwizard jvm instrumentation
• improve control of resources to collect metrics with path include/exclude path templates
• allow services to add labels to http request metrics (e.g. the GraphQL service can add the label query to specify the executed query)
• remove the json format and use only the prometheus format

Rationale

Allow administrator to better monitor RESTHeart instances and developers to snoop performance bottlenecks.

Detailed documentation

Tutorial

Run restheart enabling metrics and specifying a configuration:

docker run --rm -p "8080:8080" -e RHO="/http-listener/host->'0.0.0.0';/mclient/connection-string->'mongodb://host.docker.internal';/ping/uri->'/acme/ping';/requestsMetricsCollector/enabled->true;/jvmMetricsCollector/enabled->true;/requestsMetricsCollector/include->['/{tenant}/*']" softinstigate/restheart-snapshot:644eeab

Let's analyze the metrics configuration. With the given RHO env variable, it is:

requestsMetricsCollector:
  enabled: true
  uri: /metrics
  include:
  - /{tenant}/*
  exclude:
  - /metrics
  - /metrics/*
jvmMetricsCollector:
  enabled: true

Metrics will be collected for requests matching the include path templates and not matching the exclude ones.

Note that a variable such {tenant} in the include path templates make the metrics being tagged with path_template_param_tenant =<value>. This does not apply for wildcards in path templates.

Let's make a few requests to /acme/ping with httpie

$ http -b :8080/acme/ping
Greetings from RESTHeart!

$http -b :8080/acme/ping
Greetings from RESTHeart!

$http -b :8080/acme/ping
Greetings from RESTHeart!

Now we can ask for available metrics:

$ http -b -a admin:secret :8080/metrics
[
    "/jvm",
    "/{tenant}/ping"
]

Let's get the metrics for requests matching "/{tenant}/*":

$ http -b -a admin:secret :8080/metrics/{tenant}/\*

(omitting many rows)
http_requests_count{request_method="GET",path_template="/{tenant}/*",response_status_code="200",path_template_param_tenant="acme",} 3.0

The response is in prometheus format. The highlighted row is the metrics http_requests_count with value 3 and the following tags:

request_method="GET"
path_template="/{tenant}/*"
response_status_code="200",
path_template_param_tenant="acme"

Use prometheus to display metrics

Define the following prometheus configuration file prometheus.yml

global:
  scrape_interval:      5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
  evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.

scrape_configs:
  - job_name: 'restheart http /{tenant}/*'
    static_configs:
    - targets: ['host.docker.internal:8080']
    metrics_path: '/metrics/{tenant}/*'
    basic_auth:
      username: admin
      password: secret
  - job_name: 'restheart jvm'
    static_configs:
    - targets: ['host.docker.internal:8080']
    metrics_path: '/metrics/jvm'
    basic_auth:
      username: admin
      password: secret

Run prometheus with:

$ docker run --rm --name prometheus -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

Prometheus will start scraping restheart metrics. Note that given the default exclude path templates, metrics for prometheus requests will not be collected.

Open localhosts:9090 with your browser and check the metrics:

Add metrics labels from a Service

The method Metrics.attachMetricLabels(Request<?> request, List<MetricLabel> labels) allows to add custom labels to the collected metrics.

This is used by GraphQLService that adds to the metrics the query label with the name of the GraphQL executed query.

[ujibang]

documented at https://restheart.org/docs/monitoring