gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.61k stars 1.76k forks source link

Automatic HTTP client/server OTel Span Naming is too high cardinality #43083

Open strideynet opened 4 months ago

strideynet commented 4 months ago

Teleport uses the otelhttp library to automatically instrument our HTTP servers and clients with OTel tracing. Unfortunately, by default, it appears to use the entirety of the URL path when naming the spans. This makes it much harder to use the observability data and can cause performance issues with some observability platforms.

It would be preferable if instead each had a unique name that grouped requests based on their action, with the actual parameters being recorded as span attributes. For example, name the span DELETE /v2/thing/:thing_id rather than DELETE /v2/thing/aabbccdd.

This is backed up by the otel best practices

The span name concisely identifies the work represented by the Span, for example, an RPC method name, a function name, or the name of a subtask or stage within a larger computation. The span name SHOULD be the most general string that identifies a (statistically) interesting class of Spans, rather than individual Span instances while still being human-readable. That is, “get_user” is a reasonable name, while “get_user/314159”, where “314159” is a user ID, is not a good name due to its high cardinality. Generality SHOULD be prioritized over human-readability.

strideynet commented 4 months ago

CC @rosstimothy