[META] Enable core dynamic configuration capabilities

elastic / elastic-otel-java

Apache License 2.0

14 stars 11 forks source link

[META] Enable core dynamic configuration capabilities #86

Closed jackshirazi closed 2 months ago

jackshirazi commented 7 months ago

This is mainly for tracking, the intention is to enable these in the Otel agent itself rather than in our distribution. The list of features we'd like to be dynamically adjustable are:

instrumentation (individually enabled/disabled)
log level
enabling/disabling the agent
sending traces, metrics
sampling percentage

jackshirazi commented 7 months ago

instrumentation (individually enabled/disabled)

This has been discussed in the Otel SIG with 4 approaches: suppression, sampling, re-configuring, retransforming

Retransforming is done in the elastic agent, but is currently considered highly difficult in the otel agent
Re-configuring is a long term goal (dynamically recreating the whole agent config and applying that) but would be a huge amount of work, so is not in scope here
Sampling could work (there is already a jaeger dynamic sampler), but the instrumentation scope is not available to the sampler so it's not straightforward to do this on a per-instrumentation basis
Suppression is being looked at

jackshirazi commented 7 months ago

enabling/disabling the agent

Note that the agent being enabled is not dynamic in the Elastic agent, and the equivalent OTEL_JAVAAGENT_ENABLED non-dynamic setting is already available. What this actually refers to is recording. Since the instrumentation enabling/disabling is likely to be suppression rather than instrumentation removal, recording on/off in this situation is equivalent to disabling all instrumentation and metrics, so it's dependent on the suppression for part of the feature. A simpler approach is to set sampling to 0, but this also doesn't target metrics, and would leave the agent doing a lot more work than turning all instrumentation off. Metric disablement needs to be considered

jackshirazi commented 7 months ago

sending traces, metrics

This is leaving the agent and instrumentation enabled, but not sending anything to the server (eg to allow log correlation, or trace propagation to continue, but with no data stored for this service). This should be easily done in the exporter?

jackshirazi commented 7 months ago

sampling percentage

Available as an extension, but we should customize/do our own implementation and add it as our sampler?

jackshirazi commented 2 months ago

Closing this, next steps covered in https://github.com/elastic/elastic-otel-java/issues/300