Support Prometheus as a Sink (pull/scrape model)

[dlvenable]

Create a Sink which provides metrics to Prometheus as an API so that Prometheus can scrape Data Prepper to get metrics.

[ashoktelukuntla]

Is your feature request related to a problem? Please describe.

Data Prepper can output to OpenSearch, but cannot output to arbitrary Prometheus as a Sink in OTEL format

Describe the solution you'd like

Create a new sink in Data Prepper which outputs JSON to an arbitrary Prometheus endpoint.

It can output events to an Prometheus endpoint as a JSON array of events.

Example:

[{"key1": "value1", "key2": "value2"}, {"key1": "value3", "key2": "value4"}]

It could also be configured to write each new event as a JSON string on its own line. This is similar to the OpenSearch bulk API.

Example:

{"key1": "value1", "key2": "value2"}
{"key1": "value3", "key2": "value4"}

Additional Context:

• https://aws-otel.github.io/docs/getting-started/prometheus-remote-write-exporter
• https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/README.md
• https://prometheus.io/docs/prometheus/latest/installation/
• https://hub.docker.com/r/prom/prometheus/
• https://github.com/prometheus
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
• Example
  • Write : https://github.com/prometheus/prometheus/blob/release-2.45/documentation/examples/remote_storage/example_write_adapter/server.go
  • https://github.com/javiroman/jremotewrite
  • https://github.com/m3dbx/prometheus_remote_client_java

Possible user configurations:

endpoint (no default): The remote write URL to send remote write samples insecure (default = false) cert_file (no default): path to the TLS cert key_file (no default): path to the TLS key to use

The following settings can be optionally configured:

external_labels: map of labels names and values to be attached to each metric data point headers:. additional headers attached to each HTTP request.

• Note the following headers cannot be changed: Content-Encoding, Content-Type, X-Prometheus-Remote-Write-Version, and User-Agent. namespace: prefix attached to each exported metric name. remote_write_queue: fine tuning for queueing and sending of the outgoing remote writes. -enabled: enable the sending queue -queue_size: number of Data Prepper events that can be queued. Ignored if enabled is false -num_consumers: minimum number of workers to use to fan out the outgoing requests. resource_to_telemetry_conversion -enabled (default = false): If enabled is true, all the resource attributes will be converted to metric labels by default.

[dlvenable]

There is some initial work for this in the prometheus-sink project. But, we would need somebody to pick this up to make progress on this code to make this a usable feature.