Create specification for OpenTelemetry span attributes for Elastic-specific APM features

[SylvainJuge]

In order to provide compatibility with Elastic-only features through OpenTelemetry (either API, API+SDK, API + agent bridge or OpenTelemetry agent), we should define the span attributes and a namespace prefix for that purppose.

For example, the span.context.destination.service.resource value can be controlled by end-user through agent APIs (specific to Elastic) but we can't do the same when using the OpenTelemetry API, hence forcing users to choose between a vendor-neutral API or having the flexibility to customize the captured data.

Proposal for attribute prefix: elastic.otel.

We can split the properties in 2 categories:

• agent behavioral attributes: those that only have an effect on the agent, for example the ability to mark a span as non-discardable introduced in Java-agent in https://github.com/elastic/apm-agent-java/pull/2632 with elastic.otel.nondiscardable attribute, those are not sent to APM server. For those the cross-agent alignment is not strictly required and likely agent-specific, but they should use the common prefix for consistency.
• server attributes: those would be used to modify the mapping server-side with higher priority over the inferred/mapped values. For example it could include "service destination address", "service destination resource" (deprecated) or "service target" fields that some agents allow to be modified trough their respective APIs and now would be modifiable through standard OTel API or native OTLP. For this category we need to define and align on a common set of attributes definitions.

[eyalkoren]

We need to consider how proper it is to use attributes for behavioral configurations, or even using specially-prefixed attributes altogether. The major incentive for using OTel API is vendor neutrality. Using elastic.otel may be conceived as defying that. With behavioral attributes, this may be specifically more problematic, as we are going to act on those, but not send them as part of the spans metadata. However, if a user switches to another OTel-compliant implementation, such attributes will be added and sent as data. Not sure if this should hold us back, but worth the discussion.

[SylvainJuge]

The only example I've found so far of a non-standard attribute added to OTel spans can be found in Splunk OTel distribution: https://github.com/signalfx/splunk-otel-java/blob/7619f57606c173eeedb97445647068a467e39c68/docs/webengine-attributes.md#webengine-attributes.

[felixbarny]

Proposal for attribute prefix: elastic.otel.

Based on OTel naming schema recommendations,

To avoid clashes with names introduced by other companies (in a distributed system that uses applications from multiple vendors) it is recommended to prefix the new name by your company's reverse domain name, e.g. com.acme.shopname.

I suggest using co.elastic. as a prefix. Using otel seems redundant as OTel is implied for OTel attributes.

The major incentive for using OTel API is vendor neutrality. Using elastic.otel may be conceived as defying that.

I don't think that using Elastic-specific attribute names defies the whole purpose around vendor neutrality. If a users is concerned about using vendor-specific functionality, they can just choose to not use it.

Whenever they make sense across OTel implementation, we should, however try to contribute attributes and API changes to upstream OTel. In this case, however, it seems like a very specific feature that not even a quorum of the Elastic agents support.

However, if a user switches to another OTel-compliant implementation, such attributes will be added and sent as data.

While that is a side-effect when switching to another backend, it doesn't seem like a big issue. I wouldn't hold back because of that.

[eyalkoren]

Following the feedback above, the attribute name used in the Java agent was changed to co.elastic.discardable.