search

open-telemetry / opentelemetry-dotnet

The OpenTelemetry .NET Client
https://opentelemetry.io
Apache License 2.0
3.23k stars 765 forks source link

[bug] Divergence between metrics names exposed between Prometheus Exporter and Otlp Exporter (Unit not appended) #5814

Closed vinishiru closed 2 months ago

vinishiru commented 2 months ago

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Package Version

Package Name Version
OpenTelemetry 1.9.0
OpenTelemetry.Extensions.Hosting 1.9.0
OpenTelemetry.Instrumentation.AspNetCore 1.9.0
OpenTelemetry.Exporter.OpenTelemetryProtocol 1.9.0
OpenTelemetry.Exporter.Prometheus.AspNetCore 1.9.0 -beta.2

Runtime Version

net8.0

Description

Our application uses metrics exposed by the AddAspNetCoreInstrumentation. Before updating to the OpenTelemetry Exporter, we were using the AddPrometheusExporter + MapPrometheusScrapingEndpoint, so Prometheus could scrape the application directly.

After changing to the OpenTelemetry Exporter (AddOtlpExporter), we noticed that metric names have changed. The unit is not been appended on metric name as before, impacting the dashboards provided by .NET Team for Grafana ASP.NET Core Dashboard (You can check that the dashboard template is configured with metrics that contains the unit suffix)

Example of metric name with Prometheus Exporter: http_server_request_duration_seconds

Metric name with OtlpExporter: http_server_request_duration

Steps to Reproduce

  1. Create a OpenTelemetry Collector using docker and configure it with prometheus exporter endpoint.
  2. Create a Prometheus instance using docker and add a job to scrap metrics from Opel Collector using the previous endpoint.
  3. Create .NET ASP.NET project with the related Nuget packages.
  4. Configure the service collection with the WithMetrics extension and add both AddOtlpExporter and point it to the Opel Collector (default: http://localhost:4317).
  5. Navigate to the sample controller and check prometheus for the scrapped metrics.

Expected Result

Metric names should contains unit appended to their names. Ex:

# TYPE http_server_request_duration_seconds histogram
# UNIT http_server_request_duration_seconds seconds
# HELP http_server_request_duration_seconds Duration of HTTP server requests.

Actual Result

Metrics with no unit appended, for example:

# HELP http_server_request_duration Duration of HTTP server requests.
# TYPE http_server_request_duration histogram

Additional Context

Perhas is just a configuration issue on my part, but I could not find any configuration that makes the OpenTelemetry exporter to append the metric unit.

cijothomas commented 2 months ago

Expected Result Metric names should contains unit appended to their names.

That is, not something OTel spec supports today. It is a feature of OpenTelemetry Prometheus exporter only, and not when using any other exporter, like OTLP. If you are using Collector, then I expect collector would do the appending on name with unit when exposing to Prometheus. Or if you are using native OTLP Ingestion in Prometheus, then prometheus itself could be doing that.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md shows the OTEL and Prometheus compatibility. It is still experimental, and so is the PrometheusExporter shipped from this repo.