Jaeger tracer not resolving

[alvinhom]

I am getting an strange error in one of my environments that it fails to resolve the Jaeger tracer. I tried both 1.5.8 and 1.5.9 versions.

I am using dynamic attach with the following parameters:

-Dsa.tracer=jaeger -Dsa.log.level=FINEST -DJAEGER_REPORTER_LOG_SPANS=true -Dsa.log.file=/tmp/specialagent.log -DJAEGER_AGENT_HOST=172.22.163.157 -DJAEGER_AGENT_PORT=6831 -DJAEGER_SAMPLER_TYPE=probablistic -DJAEGER_SAMPLER_PARAM=0.01 -Dsa.instrumentation.plugin.servlet.skipPattern=./healthcheck.|./rest/version/buildnumber|./abc/rest/about -DJAEGER_SERVICE_NAME=test

I see the following logs in the specialagent log file:

<<<<<<<<<<<<<<<<<<<< Loading Tracer Plugin >>>>>>>>>>>>>>>>>>>>>

Resolving tracer: jaeger Found META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory: java.net.URL@647ca601jar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent15129429306732819718/META-INF/plugins/otel.jar!/META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory Found META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory: java.net.URL@48365e4bjar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent15129429306732819718/META-INF/plugins/jaeger.jar!/META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory Tracer was NOT RESOLVED

Loaded Tracer Plugin <<<<<<<<<<<<<<<<<<<<<<

Clearly the tracer is set to "jaeger" and the jaeger.jar plugin is found. But it prints out that the tracer was not resolved.

Do you know what could be the cause? Classloading issues?

[safris]

Hi @alvinhom, thanks for mentioning this issue. I will take a look at it!

[alvinhom]

Hi @safris, I addded some debugging logging stmts. Here are the logs:

jaeger findTracer: 'jaeger' Found META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory: java.net.URL@5c1c46f9jar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/lightstep.jar!/META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory Found META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory: java.net.URL@d5fb6edjar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/otel.jar!/META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory Found META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory: java.net.URL@27527052jar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/jaeger.jar!/META-INF/services/io.opentracing.contrib.tracerresolver.TracerFactory Found match /opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/jaeger.jar TracerResovlerResourcePath: jar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/opentracing-tracerresolver-0.1.8.jar!/io/opentracing/contrib/tracerresolver/TracerResolver.classtracerResolverResourceUrl URL: jar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/opentracing-tracerresolver-0.1.8.jar!/io/opentracing/contrib/tracerresolver/TracerResolver.class tracerURL:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/jaeger.jar TracerResolverClass URL:jar:file:/opt/dwp/apache/tomcat/temp/opentracing-specialagent3147712352103026824/META-INF/plugins/opentracing-tracerresolver-0.1.8.jar!/io/opentracing/contrib/tracerresolver/TracerResolver.class Tracer was NOT RESOLVED

So, it appears that the jaeger jar file is found, as shown by the tracerURL that is returned. However, it seems the class is actually loaded by this

    AgentRuleUtil.tracerClassLoader = new URLClassLoader(new URL[] {tracerResovlerUrl}, parent);

Which is resolving using the opentracing-tracerresolver-0.1.8.jar, which is not the jaeger one.

Also, the strange thing is that this breaks only in one of my environments and works on others. I am not sure whether it is related on the order that classes are loaded in the PluginsClassLoader. I see that in the debugging statements, the opentracing-tracerresolver-0.1.8.jar is higher up in the class loading order than jaeger.jar.

Hope that helps.

[safris]

Hi @alvinhom, SpecialAgent v1.6.1 has just been released. Can you test your use-case and let me know if this issue is fixed (or still outstanding)?!

[alvinhom]

@safris Thanks. My test environment is being rebuild, so it will take a couple of day. I will retest and update the thread.

[safris]

Hi @alvinhom, we have another release in a couple days, so I'm going to close this issue. If you discover that Jaeger is still not resolving correctly, please feel free to reopen this issue.

[alvinhom]

@safris Thanks. I forgot to update the issue. With the latest 1.7.1, I am no longer able to reproduce the issue. So it looks like it was fixed. Thanks again.