[bug] Duplicated Attribute Values in Metrics

[Coelho04]

Package

OpenTelemetry

Package Version

Package Name	Version
OpenTelemetry	1.9.0
OpenTelemetry.Exporter.Console	1.8.0
OpenTelemetry.Exporter.OpenTelemetryProtocol	TBD
OpenTelemetry.Extensions.Hosting	TBD
OpenTelemetry.Extensions.Propagators	TBD
OpenTelemetry.Instrumentation.AspNetCore	TBD
OpenTelemetry.Instrumentation.Http	TBD
OpenTelemetry.Instrumentation.Process	TBD
OpenTelemetry.Instrumentation.Runtime	TBD

Runtime Version

net8.0

Description

We are seeing the http_client_requestduration(sum,count,bucket) attributes with multiple values for instance:
http_client_request_duration_count{environment="prd", error_type="400;400", http_request_method="POST;POST", http_response_status_code="400;400", instance="instance_x", job="application_x", network_protocol_version="1.1;1.1", server_address="web_api1;web_api1", url_scheme="https;https"}

We have several applications with the same configurations regarding OpenTelemetry, but for some reason, only a few have this problem, and we can't understand why.

The applications receive high throughput around 100k rpm.

Only this metric seems to be affected.

We believe that the issue might be when in a custom HttpHandler we perform another call to another API to get an authorization token but we have yet to validate this.

Can someone take a look at this?

Steps to Reproduce

We don't have the certainty that this can reproduce the issue but might help investigate the issue.

• Create an application that needs to connect to some API
• Create and handler that will fetch a token or make another call to the same or other API.
• After receiving the call from that the other API let the handler continue with the other request.

Expected Result

http_client_request_duration_count{environment="prd", error_type="400", http_request_method="POST", http_response_status_code="400", instance="instance_x", job="application_x", network_protocol_version="1.1", server_address="web_api1", url_scheme="https"}

Actual Result

http_client_request_duration_count{environment="prd", error_type="400;400", http_request_method="POST;POST", http_response_status_code="400;400", instance="instance_x", job="application_x", network_protocol_version="1.1;1.1", server_address="web_api1;web_api1", url_scheme="https;https"}

Additional Context

No response

[cijothomas]

What is the .NET version used? In newer ones, the metrics are produced by the httpclient library itself, but in older ones the instrumentation library in OTel does that. Once we know who is adding duplicate tags, we can decide next steps on the fix.

[Coelho04]

@cijothomas we are using .net8.
From my understanding, the metrics are produced by the MetricsHandler.cs that's present in System.Net.Http.Metrics.