Enhance ECS logging correlation fields

[SylvainJuge]

• [x] specification: add service.environment in ECS reformatting: https://github.com/elastic/apm/pull/766
• [x] specification: define "service-level" correlation and "trace-level" correlation in specs/agents/log-correlation.md : https://github.com/elastic/apm/pull/766
  • trace level is defined with trace.id, transaction.id
  • service level is defined with service.name, service.version, service.environment
  • [x] question: should we include the container.id in the agents specification too ? Yes & no: we can mention it for reference but agents are not expected to set it.
• [x] documentation: document the log correlation fields (both service-level and trace-level) in the "log-correlation" section:
  • APM: https://www.elastic.co/guide/en/apm/guide/current/log-correlation.html#_add_apm_identifiers_to_your_logs : https://github.com/elastic/apm-server/pull/10454
  • Application logs: https://www.elastic.co/guide/en/observability/current/application-logs.html provide filebeat|ecs-logging configuration options to set the service-level metadata. https://github.com/elastic/observability-docs/pull/2731
  • for each agent, here for Java : https://www.elastic.co/guide/en/apm/agent/java/current/logs.html#log-correlation-ids (we might also take the opportunity to also align the Logs documentation structure of each agent) #800
• [x] add service.environment to ECS logging documentation: https://www.elastic.co/guide/en/ecs-logging/overview/current/intro.html#_configurable_fields : https://github.com/elastic/ecs-logging/pull/75

Out of scope for now

• spec and document behavior of error.id when an error is reported in the logs (currently the Java agent creates an error, this feature is currently documented as Error capturing)
• adding service.name, service.environment and service.version to the fields that are automatically injected in MDCs for log correlation: this would allow for example to use those in plain-text logs.
• specify and document the span.id for log correlation.

[felixbarny]

question: should we include the container.id in the agents specification too ?

I think it doesn't hurt mentioning it but there's nothing that agents need to do as they already send the container.id as part of the metadata. The container.id correlation is more relevant when shipping the logs with Filebeat and when there's no service.name within the logs.

documentation: document the log correlation fields (both service-level and trace-level) in the "log-correlation" section:

• APM: https://www.elastic.co/guide/en/apm/guide/current/log-correlation.html#_add_apm_identifiers_to_your_logs

+1 on improving the docs to point out how service/log correlation works and how to configure Filebeat or Elastic Agent to add service.name/service.environment metadata. The guide also focusses mainly on the semi-deprecated Filebeat and misses to mention Elastic Agent.

[SylvainJuge]

I've added a mention of container.id in https://github.com/elastic/apm/pull/766/commits/6b9968f8a21a81a85ba8c7820f082dc76f771082

+1 for the documentation part, I will add it to the shopping list in this issue description, same for providing the configuration options to set those fields in the filebeat configuration and maybe in ECS-logging. When the APM agent is combined with ECS-logging library I would recommend to let the agent provide its own values though to avoid inconsistencies.

For the fact that Elastic Agent isn't mentioned, I think it would be better handled as a follow-up once this part is done. I haven't tried using "Custom logs" integration yet, but that should not be very different from the filebeat configuration that we currently offer.

[SylvainJuge]

Closing as specs and documentation are complete, implementation in agents is not 100% complete for service environment correlation see https://github.com/elastic/apm/issues/773 for details.