Meaningful operation names when using framework integrations

[objectiser]

When OpenTracing API is used explicitly by an application it can provide meaningful operation names.

However in cases where framework integrations are used, to avoid adding tracing code into the business logic, it can be difficult (and sometimes impossible) for any meaningful operation name to be determined.

A prime example is many of the HTTP/REST instrumentations where only a HTTP method and URL are available. The URL could potentially be used, but if it contains path parameters, this could lead to a large set of operation names actually related to the same endpoint. Similarly the HTTP method is not particularly useful, as there could be many endpoints using the same HTTP method on the service - and therefore does not distinguish the endpoint suitably.

With server spans, it is possible that the application code (or another layer of framework integration) could be used to override the operation name already set by some lower level framework integration. However the same cannot be achieved for client spans, as the higher level that may have access to a suitable operation name does not have access to the client span created in the lower level (framework integration).

There are a number of different approaches that could be used:

1) Deal with this at the framework integration levels - some frameworks may provide ways in which more meaningful operation names could be derived, but the quality of name would vary across integrations.

2) Deal with server spans as now - access it as the active span to override its operation name. For the client side, provide some 'hint' mechanism for client spans, where the higher level code can pre-initialize operation name (and possibly tags) which then get inherited when the client span is created, as mentioned here: https://github.com/opentracing/opentracing-java/pull/115#pullrequestreview-36748848

3) Provide a general solution that uses initial information provided when a Span is created (e.g. combination of http.url, http.method and span.kind) to identify an operation name (and optionally other tags). To overcome URLs with path parameters, regular expressions could be used. The benefit of such an approach is that it could be generally applied across client and server spans, and also supported across all languages, leading to consistent operation naming. However it does incur an admin overhead - but the underlying operation name resolution mechanism could be pluggable, allowing tracer vendors to provide mechanism to help manage this mapping data.

[tedsuo]

@objectiser thanks for starting this. I think 1) is the correct solution in the sense that "frameworks" is a very broad category. For example, control flow frameworks may all have the same semantics, so what makes a good operation name will differ from framework to framework.

However, I think frameworks that implement the same protocols should ideally share some conventions, so that you can mix and match client/server libraries and not have that radically change the kind of naming conventions you have available. And HTTP is definitely the protocol we should provide guidance on... perhaps just start with that? Other protocols can use the HTTP naming conventions as a guide to best practices.

[objectiser]

Agree (1) is the simplest option - but the problem is the information these framework integrations have access to - and in the case of HTTP, this is generally the HTTP method and URL.

If we can come up with some guidance based on that limited information, then that would be great - any suggestions that avoid the issue with path parameters?

[tedsuo]

I think route-names/path-parameters are a good example of how frameworks can have semantics that are not shared universally. I agree that if you have such a concept, you should use it in your framework instrumentation, and possibly forgo instrumenting at the HTTP/protocol level since it would be almost identical spans.

But if you are instrumenting at the HTTP/protocol level, there is a standard you could adhere to. My first draft would be:

• add the standard RPC tags
• add the tags http.method, http.url, http.method and http.status_code.
• Provide an option to use either http.url or http.method as the operation name. One has a cardinality that is too high, the other too low. Pick your poison.
• Provide a reasonable interceptor abstraction so that frameworks which are using the protocol library can override method name with something more semantically meaningful

[jakerobb]

Perhaps OT should expose a way to set the operation name on the current (already started) span?

This would be most useful in situations in which the information you really want to use is not readily available at the process boundary.

[objectiser]

@tedsuo Your last bullet is essentially equivalent to my option (3) - although I discussed a specific way that such an interceptor could determine a meaningful name.

@jakerobb It is possible to change the span's operation name after start (atleast in the Java API).

[pavolloffay]

@jakerobb operation name can be set on an already started span. see https://github.com/opentracing/opentracing-java/blob/master/opentracing-api/src/main/java/io/opentracing/Span.java#L163, but you have to have an access to the current span.

Operation name should have some semantical meaning/definition. When using URL the cardinality can be too high card(operation X) = total number of spans of operation X.

I think this issue comes from the fact that in most tracing systems an operation name is the only string displayed with a span. I would rather see a clarification of what it should semantically represent.

[mabn]

I would definitely advise against using URL as operation name by default as they often contain identifiers. Trying to aggregate data based on those names becomes time-consuming, imperfect and often leads to crazy regexps. Especially when some outliers appear only after parsing terabytes of logs.

Regarding client-side: Usually when I call some service I know what I'm calling and I'd prefer a way to provide that name. When using hand-written (or generated) API clients the names of operations can be embedded in them. Fallback to "http_get" is perfectly fine.

Many http clients provide some way to pass extra parameters to lower layers. It would be good to have such option for cases where it's not possible. One object is already passed this way - SpanContext. Maybe it would be possible to piggyback such extra details on the same mechanism.

[yurishkuro]

Our RPC team has developed a framework yarpc that allows abstracting the underlying transport from application code. The requests are always required to specify service and endpoint name (which makes it very easy to define tracing Span). And for HTTP transport, those values are always set as HTTP headers (here). Just fyi.