search

elastic / apm-agent-java

https://www.elastic.co/guide/en/apm/agent/java/current/index.html
Apache License 2.0
567 stars 320 forks source link

make log_sending=true correlate to the correct service #3123

Open wolframhaussig opened 1 year ago

wolframhaussig commented 1 year ago

Is your feature request related to a problem?

This issue is based on my post on Discuss: https://discuss.elastic.co/t/room-for-improvement-for-log-sending-true/330602/1

We have a Tomcat container running with multiple applications. Each application is a Spring webapplication deployed as a War-file. we use the apm agents service-name-auto-detection to get the correct service.name for each app. this looks like this:

When activating log_sending, the logs are transferred without errors to the Elastic Stack, but the Logs are not correctly correlated to the matching service. Instead, we see the following:

The agent seems to instrument the logback appenders and has no reference to the original application (event.dataset is set to tomcat-application.ApplicationAAppenderDev, service.name is tomcat-application). This means, calling https://my-kibana-host:5601/s/my-team/app/apm/services/ApplicationA/logs does not show the application logs.

Describe the solution you'd like

I would prefer a solution where the log_sending logic detects the correct service.name the same way as the traces are generated. So I would expect this in Kibana APM UI:

Describe alternatives you've considered

As a workaround we could filter based on the event.dataset which seems to be set to the service and logging-appender name. This works for correlating data manually as we have specially named appenders but this will not show the Logs correctly in the Kibana APM UI for the correct service.

Additional context

Kibana: 8.7.0 APM integration: 8.7.0 APM Java Agent: 1.36.0

JonasKunz commented 1 year ago

I suspect this issue also applies to log reformatting and log correlation. IINM, we don't handle multi-service deployments within the same JVM there yet.

I think the correct solution would be to discover the service name based on Thread.currentThread().getContextClassLoader() at the time the log event is generated.