Ability to customize internal telemetry

[yurishkuro]

Is your feature request related to a problem? Please describe.

Jaeger v2 is built as a custom distribution of collector that combines ingestion capabilities with Jaeger Query & UI extension. Historically we allowed 100% trace sampling for all query endpoints, and none for the receiver endpoints (because tracing receiver endpoints creates a potential for infinite recursive loop of traces).

Now that we're looking at enabling internal telemetry in collector, the TracerProvider is initialized exclusively by collector core and passed to components. Even if the configuration allows to configure a sampler, it would be the same sampler for receivers and extensions.

We need an ability to differentiate how traces are produced for receivers and extensions.

Describe the solution you'd like

There can be multiple options:

• extensive sampler configuration that can vary settings by endpoints
• or just a hook in the core API to customize how providers are created, e.g. a way to inject a custom sampler (for which there may be no configuration support for a while)

Describe alternatives you've considered

We have an option of ignoring TracerProvider passed from the core in some of our components. E.g. we can instruct the user not to configure service: telemetry: traces: (so that Noop provider is passed by default) and instantiate our own provider in the extensions.

[codeboten]

I wonder if we could use something like level https://github.com/open-telemetry/opentelemetry-collector/blob/fbadc23b71b350343226c7d707fc7d843c05650d/cmd/mdatagen/metadata-schema.yaml#L134 configuration in the short term to instantiate a no-op tracer, similarly to how level is used today for metrics in the telemetry builder https://github.com/open-telemetry/opentelemetry-collector/blob/fbadc23b71b350343226c7d707fc7d843c05650d/exporter/exporterhelper/internal/metadata/generated_telemetry.go#L97

Could be configurable via something like this:

receivers:
  otlp:
    telemetry:
      spans:
        level: none

Would a no-op tracer work for this use case or would this need to be more granular than either on or off for the entire component?

extensive sampler configuration that can vary settings by endpoints

Supporting multiple samplers would require supporting multiple tracer provider configuration and allowing components to be configured to use specific ones, this isn't currently supported in the configuration schema, should it be?

[codeboten]

Alternatively, from a configuration standpoint, could we think of components in the collector as we do instrumentations in SDKs and make them configurable that way?

[yurishkuro]

Would a no-op tracer work for this use case or would this need to be more granular than either on or off for the entire component?

Right now it would work if it's on/off for the entire component: off for receivers, on for extensions (then again, should /metrics endpoint be traced? I don't think so).

Alternatively, from a configuration standpoint, could we think of components in the collector as we do instrumentations in SDKs and make them configurable that way?

Theoretically, yes, but in practice the instrumentation is generally 3rd party, like grpc-otel, which takes Provider and gets Tracer with "grpc-otel" scope. So from the SDK perspective it gets no signal which tracer is for receiver gRPC endpoint and which one for an extension's gRPC endpoint. In theory it's still achievable if scope attributes were different (where one of the attributes cold be otelcol.component). But again, the existing APIs and instrumentations don't really make this easy to define / override. And even if they did, there's no configuration mechanism to instruct provider how to create tracers differently for different scopes.

[yurishkuro]

can this issue be marked as a blocker for v1 / GA?