opentracing-contrib / java-specialagent

Automatic instrumentation for 3rd-party libraries in Java applications with OpenTracing.
Apache License 2.0
185 stars 46 forks source link

Tracing requests to Elasticsearch with the specialagent? #597

Open jgoeres opened 4 years ago

jgoeres commented 4 years ago

Hi,

I managed to configure the specialagent for most microservices in our stack and I am seeing cross-service traces now. Alas, one important component in our stack is Elasticsearch, and requests to it do not show up at all. Our microservices use the Apache HTTP Client and/or the Elasticsearch high level client, so I have the two instrumentations enabled for those, i.e.

    -Dsa.instrumentation.plugin.apache:httpclient.enable 
    -Dsa.instrumentation.plugin.elasticsearch:*.enable 

(Note: these are the 1.6.x properties, since I am having trouble with 1.7.x due to https://github.com/opentracing-contrib/java-specialagent/issues/589) I assumed that this is because I have not instrumented ES itself (even though I had hoped to see the outbound requests as they are being sent by the microservice). So I tried adding the specialagent to ES, which didn't work too well, as ES's security manager gets in the way:

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createSecurityManager")
   at java.base/java.security.AccessControlContext.checkPermission(Unknown Source)
   at java.base/java.security.AccessController.checkPermission(Unknown Source)
   at java.base/java.lang.SecurityManager.checkPermission(Unknown Source)
   at java.base/java.lang.SecurityManager.<init>(Unknown Source)
   at io.opentracing.contrib.specialagent.AgentRuleUtil$CallingClass.<init>(AgentRuleUtil.java:101)
   at io.opentracing.contrib.specialagent.AgentRuleUtil$CallingClass.<init>(AgentRuleUtil.java:101)
   at io.opentracing.contrib.specialagent.AgentRuleUtil.getExecutionStack(AgentRuleUtil.java:119)
   at io.opentracing.contrib.specialagent.AgentRule$1.childValue(AgentRule.java:100)
   at io.opentracing.contrib.specialagent.AgentRule$1.childValue(AgentRule.java:89)
   at java.base/java.lang.ThreadLocal$ThreadLocalMap.<init>(Unknown Source)
[...]

I am seeing this as soon as I enable any instrumentation for ES, only if I disable them all (but still add the special agent), ES will start. Alas, it seems that disabling the security manager in ES, or using a relaxed security policy, is not supported anymore.

Does anyone have a proposal how to get ES requests to show up in traces? Client-side would be fine, but if there is a way to instrument ES itself, that would also be interesting.

THX J