Come up with guidance on how to add experimental APIs to stable signals

[tigrannajaryan]

As existing stable signals in Otel mature we are likely to see the need to evolve these signals such that new APIs are added to them first in unstable form and then are stabilized. We currently don't have any guidance on how this should be achieved and here is for example a case where an attempt to add an experimental implementation becomes difficult because language maintainers would like to have a stable spec before accepting an implementation, however we have a chicken and egg problem here since we don't allow the spec to stabilize before there are implementations in several languages.

I believe we need to come up with a process that explains how these experimental additions to existing stable APIs are done in the spec and in the language implementations. One possible way is to leverage the recently added (but not widely used in the spec) more granular maturity levels. Language implementations will likely also need to find a way to bring unstable additions to existing signals (some implementations already have this ability, e.g. Java).

This issue is a request for comments: language maintainers and spec sponsors please tell what you think about the need for such guidance and if you have ideas please make proposals.

cc @open-telemetry/technical-committee @open-telemetry/spec-sponsors

[mx-psi]

I think this is going to be significantly different depending on the language and the tools available there. For example, Rust features are typically used for this kind of experimental APIs while Golang build tags (the closest equivalent concept in Go) are not typically used in this fashion in the Go ecosystem and may be stranger to use for end-users.

For Go in particular the Collector SIG may be able to provide ideas here, e.g. the recent work on adding profiling as well as part of the 1.0 work has involved a lot of "figuring out how to isolate experimental bits from other parts already marked as 1.0/scheduled to be marked as 1.0". Some tools we have used are optional interfaces that may be implemented by a struct that is presented through a more generic interface (e.g. component.Host is the generic interface while componentstatus.Reporter is the concrete, experimental one) or re-exporting the API of internal packages into two different modules (one stable, one experimental, a simple example here is the constants in pipeline vs the ones in componentprofiles). For some of this work we have collaborated with @dmathieu so I am sure the Go SIG is aware of this, but maybe we can put this into writing :)

[jack-berg]

@tigrannajaryan does this belong in opentelemetry-specification?

[tigrannajaryan]

@tigrannajaryan does this belong in opentelemetry-specification?

I thought this is not just a spec issue but also impacts language implementations. We can move to the spec if you feel that's a better place for this discussion.

[jack-berg]

No strong preference - just wasn't sure if it was intentional.

[dmathieu]

I agree with @mx-psi that the behavior really depends on the language, and what it offers. Go is rather strict there. An interface can't add a new method, that'd be a breaking change for any implementations of that interface (the Go API solved that with embedded/trace/noop interfaces, but that can't be done for the SDK). So we have to implement new interfaces that define the new behavior (see the WIP OnEnding implementation).

In Ruby for example, it's easy to make new interface methods a trivial change with the use of respond_to (see their implementation of OnEnding).

While we can definitely provide some guidance or ideas, this is so dependent on the language that it seems difficult to provide a general way things should be done.