[Discuss] Listen to OpenTelemetry environment configuration by default

[Mpdreamz]

Relates to: https://github.com/elastic/apm/pull/813 where we'd listen to OTEL_* environment variables in the context of logging.

If we agree to make our ecs-logging OpenTelemetry aware it makes sense to extend the same logic to our agent configuration.

Quick summary is to propose adding support for the following OTEL_* environment variables.

• OTEL_SDK_DISABLED
• OTEL_RESOURCE_ATTRIBUTES
• OTEL_SERVICE_NAME
• OTEL_LOG_LEVEL

This would create a new resolving logic for the listed APM Agent configuration items:

• Enabled

  • ELASTIC_APM_ENABLED
  • the negation of OTEL_SDK_DISABLED if set
  • defaults to true

• ServiceName

  • ELASTIC_APM_SERVICE_NAME
  • OTEL_SERVICE_NAME
  • OTEL_RESOURCE_ATTRIBUTES value for service.name
  • defaults to agents current service name detection defaults.

• ServiceVersion

  • ELASTIC_APM_SERVICE_VERSION
  • OTEL_RESOURCE_ATTRIBUTES value for service.version
  • defaults to agents current service version detection

• Environment

  • ELASTIC_APM_ENVIRONMENT
  • OTEL_RESOURCE_ATTRIBUTES value for deployment.environment
  • defaults to agents current environment detection

• ServiceNodeName

  • ELASTIC_APM_SERVICE_NODE_NAME
  • OTEL_RESOURCE_ATTRIBUTES value for service.instance.id
  • defaults to agents current environment detection

• LogLevel

  • ELASTIC_APM_LOG_LEVEL
  • OTEL_LOG_LEVEL
  • defaults to Error.

We could later follow this up with a discussion on what other resource semantic conventions we could support in the future. As well as finding a way to map OTEL_RESOURCE_ATTRIBUTES to GlobalLabels but leaving it out of scope for now.

[basepi]

This seems reasonable to me.

[jackshirazi]

We're currently trying to implement the Java agent as follows:

• the agent is an extension of the Opentelemetry agent, so ALL the Opentelemetry options apply
• the agent will support the existing elastic options as default, so whatever default we specify and whatever override the user uses is also default
• where the elastic option matches an Opentelemetry option, per the ones above, the Opentelemetry option if used sets the final value

So for example, take the log level. The Elastic default is INFO. So

1. the default is set to the Elastic default (INFO)
2. if ELASTIC_APM_LOG_LEVEL is set, the value is set to that
3. if OTEL_LOG_LEVEL is set, then that provides the final value for the log level (overriding the ELASTIC_APM_LOG_LEVEL if that was set)

[jackshirazi]

Note that we rolled back trying to support elastic config in the Java agent. The current strategy is to not explicitly support any elastic configs and give documentation that tells users how to convert from a subset of elastic configs to otel configs

[Mpdreamz]

Thanks for highlighting that @jackshirazi 👍 We'll follow on from your learnings and won't do this in our agent, existing or new OTEL wise.

listening to OTEL environment variables in our logging libraries might be its own discussion though as per: https://github.com/elastic/apm/issues/816

Will put that back up on the agenda next week.