zio / zio-metrics-connectors

Monitoring, Metrics and Diagnostics for ZIO
https://zio.dev/zio-metrics-connectors
Apache License 2.0
30 stars 24 forks source link

Follow Prometheus Exposition format in Prometheus Encoder #34

Closed tup916 closed 1 year ago

tup916 commented 1 year ago

When I connect the prometheus published metrics from my service which includes DefaultJvmMetrics to Telegraf, I see several parsing errors as follows:

2023-03-09T16:31:20Z E! [inputs.prometheus] Error in plugin: error reading metrics for http://localhost:9000/metrics: reading text format failed: text format parsing error in line 7: second TYPE line for metric name "jvm_memory_pool_bytes_max", or TYPE reported after samples

I see the error for various metric names when using DefaultJvmMetrics, including but not limited to: jvm_memory_pool_bytes_init, jvm_memory_bytes_init, jvm_memory_pool_bytes_max, jvm_memory_bytes_max, jvm_gc_collection_seconds_sum, jvm_memory_pool_bytes_used, jvm_memory_pool_bytes_max.

I notice that this is because the Prometheus Encoder implementation does not exactly follow the exposition format (https://prometheus.io/docs/instrumenting/exposition_formats/#comments-help-text-and-type-information), specifically, the rule that "Only one TYPE line may exist for a given metric name."

The metrics with the same metric name but different labels should be combined together with exactly one TYPE line.

The actual output from DefaultJvmMetrics with prometheus publisher:

# TYPE jvm_gc_collection_seconds_count gauge
# HELP jvm_gc_collection_seconds_count
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 4.0 1678469178428
...
...
# TYPE jvm_gc_collection_seconds_count gauge
# HELP jvm_gc_collection_seconds_count
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0 1678469178428

The expected output:

# TYPE jvm_gc_collection_seconds_count gauge
# HELP jvm_gc_collection_seconds_count
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0 1678469178428
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 4.0 1678469178428