search

elastic / apm-agent-java

https://www.elastic.co/guide/en/apm/agent/java/current/index.html
Apache License 2.0
566 stars 320 forks source link

help: in the external plugin, the static class loading has java.lang.NoClassDefFoundError #1445

Closed pengpj closed 2 years ago

pengpj commented 4 years ago

Refer to integration-tests/external-plugin-test, i create an external plugin related to logback, but when calling a KafkaManager class, an exception occurs: java.lang.NoClassDefFoundError: Could not initialize class co.elastic.apm.agent.log.collect.kafka.KafkaManager

about KafkaManager 

import co.elastic.apm.agent.sdk.state.GlobalState;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer

@GlobalState
public class KafkaManager {

    private static final KafkaProducer<String, String> producer = initProducer();

    private static KafkaProducer<String, String> initProducer() {
        LogCollectEnvConfig configuration = new LogCollectEnvConfig();
        String bootstrapServers = configuration.getBootstrapServers();

        Properties config = new Properties();
        config.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        config.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        config.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        config.setProperty("batch.size", "0");
        return new KafkaProducer<>(config);

    }

    public static void send(final String msg) {
        ...
    }

}

logback instrument call it 


import co.elastic.apm.agent.sdk.ElasticApmInstrumentation;

import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.IThrowableProxy;
import ch.qos.logback.classic.spi.StackTraceElementProxy;
import net.bytebuddy.asm.Advice;
import net.bytebuddy.description.method.MethodDescription;
import net.bytebuddy.description.type.TypeDescription;
import net.bytebuddy.matcher.ElementMatcher;

import java.util.Collection;

import static net.bytebuddy.matcher.ElementMatchers.named;

public class LogbackLogCollectorInstrumentation extends ElasticApmInstrumentation {

    private static final String LOGBACK_EVENT = "ch!qos!logback!classic!Logger".replace("!", ".");

    @Override
    public ElementMatcher<? super TypeDescription> getTypeMatcher() {
        return named(LOGBACK_EVENT);
    }

    @Override
    public ElementMatcher<? super MethodDescription> getMethodMatcher() {
        return named("callAppenders");
    }

    @Override
    public Collection<String> getInstrumentationGroupNames() {
        Collection<String> groupNames = super.getInstrumentationGroupNames();
        groupNames.add("logback");
        return groupNames;
    }

    @Override
    public Class<?> getAdviceClass() {
        return LogbackAdvice.class;
    }

    public static class LogbackAdvice {

        /**
         * collect logback log info
         */
        @Advice.OnMethodEnter(suppress = Throwable.class, inline = false)
        public static void onEnter(@Advice.Argument(value = 0, optional = true) ILoggingEvent param) {
            ...
            KafkaManager.send("send some thing");
        }
    }
}

use param ELASTIC_APM_PLUGINS_DIR to loading, stack ：

java.lang.NoClassDefFoundError: Could not initialize class co.elastic.apm.agent.log.collect.kafka.KafkaManager
    at co.elastic.apm.plugin.log.collect.LogbackLogCollectorInstrumentation$LogbackAdvice.onEnter(LogbackLogCollectorInstrumentation.java:85)
    at ch.qos.logback.classic.Logger.callAppenders(Logger.java:255)
    at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
    at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
    at ch.qos.logback.classic.Logger.log(Logger.java:765)
    at org.apache.ibatis.logging.slf4j.Slf4jLocationAwareLoggerImpl.debug(Slf4jLocationAwareLoggerImpl.java:61)
    at org.apache.ibatis.logging.slf4j.Slf4jImpl.debug(Slf4jImpl.java:74)
    at org.apache.ibatis.logging.jdbc.BaseJdbcLogger.debug(BaseJdbcLogger.java:159)
    at org.apache.ibatis.logging.jdbc.ConnectionLogger.invoke(ConnectionLogger.java:53)
    at com.sun.proxy.$Proxy206.prepareStatement(Unknown Source)
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.instantiateStatement(PreparedStatementHandler.java:87)
    at org.apache.ibatis.executor.statement.BaseStatementHandler.prepare(BaseStatementHandler.java:88)
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.prepare(RoutingStatementHandler.java:59)
    at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:85)
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)
    at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
    at com.sun.proxy.$Proxy129.selectList(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
    at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:137)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:75)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59)
    at com.sun.proxy.$Proxy177.selectMetadataList(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy178.selectMetadataList(Unknown Source)
    at com....(ApmCacheServer.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

env : jdk 1.8

<parent>
        <artifactId>apm-agent-parent</artifactId>
        <groupId>co.elastic.apm</groupId>
        <version>1.18.1.RC1-SNAPSHOT</version>
    </parent>

 <dependencies>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>apm-agent-plugin-sdk</artifactId>
            <version>${project.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>apm-agent-api</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>1.1.1</version>
        </dependency>
</dependencys>

classloader info :


class-info        co.elastic.apm.agent.log.collect.kafka.KafkaManager                                                                                                                                                                
 code-source                                                                                                                                                                                                                          
 name              co.elastic.apm.agent.log.collect.kafka.KafkaManager                                                                                                                                                                
 isInterface       false                                                                                                                                                                                                              
 isAnnotation      false                                                                                                                                                                                                              
 isEnum            false                                                                                                                                                                                                              
 isAnonymousClass  false                                                                                                                                                                                                              
 isArray           false                                                                                                                                                                                                              
 isLocalClass      false                                                                                                                                                                                                              
 isMemberClass     false                                                                                                                                                                                                              
 isPrimitive       false                                                                                                                                                                                                              
 isSynthetic       false                                                                                                                                                                                                              
 simple-name       KafkaManager                                                                                                                                                                                                       
 modifier          public                                                                                                                                                                                                             
 annotation        co.elastic.apm.agent.sdk.state.GlobalState                                                                                                                                                                         
 interfaces                                                                                                                                                                                                                           
 super-class       +-java.lang.Object                                                                                                                                                                                                 
 class-loader      +-co.elastic.apm.agent.bci.classloading.IndyPluginClassLoader@4acc5dff                                                                                                                                             
                     +-co.elastic.apm.agent.shaded.bytebuddy.dynamic.loading.MultipleParentClassLoader@2d865ac                                                                                                                        
                       +-co.elastic.apm.agent.bci.classloading.ExternalPluginClassLoader@543c6f6d                                                                                                                                     
 classLoaderHash   4acc5dff                                                                                      

[arthas@1]$ classloader -c 3a320ade --load co.elastic.apm.agent.log.collect.kafka.KafkaManager
load class error.
java.lang.ClassNotFoundException: co.elastic.apm.agent.log.collect.kafka.KafkaManager
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.taobao.arthas.core.command.klass100.ClassLoaderCommand.processLoadClass(ClassLoaderCommand.java:216)
        at com.taobao.arthas.core.command.klass100.ClassLoaderCommand.process(ClassLoaderCommand.java:111)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
        at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.ClassNotFoundException: co/elastic/apm/agent/log/collect/kafka/KafkaManager
                at java.lang.Class.forName0(Native Method)
                at java.lang.Class.forName(Class.java:348)
                at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:155)
                ... 15 more

How can org.springframework.boot.loader.LaunchedURLClassLoader be loaded into KafkaManager? Please!

felixbarny commented 4 years ago

Could you share the jar and/or the source code of your plugin? Am I correct in assuming you have put this jar in a plugins folder and have configured plugins_dir to point to that?

The kafka-clients dependency is just available in the plugin's jar and not in your application, is that correct?

Could you please also share the full debug logs of the startup?

pengpj commented 4 years ago
  1. the plugin code

fork repository : https://github.com/pengpj/apm-agent-java/tree/log-send path : es-apm-agent/apm-external-plugins

env config : 

    log_collect_bootstrap_servers=10.21.17.108:9092;
    log_collect_level=INFO;
    log_collect_sync_send=true;
    ELASTIC_APM_PLUGINS_DIR=/opt/plugins-dir

start an spring demo , logging by logback , command for : 

java \ 
-Xms725M \ 
-Xmx925M \ 
-XX:+PrintGCDetails \ 
-Xloggc:/app/logs/gc.log \ 
-XX:+UnlockExperimentalVMOptions \ 
-XX:+UseCGroupMemoryLimitForHeap \ 
-XX:+PrintGCDateStamps \ 
-XX:+PrintHeapAtGC \ 
-Dfile.encoding=UTF-8 \ 
-Dsun.jnu.encoding=UTF-8 \ 
-server \ 
-javaagent:/opt/elastic-apm-agent.jar \ 
-jar \ 
demo-http.jar

no more debug log.

  1. the demo no kafka client . Use maven-shade-plugin in the build , it would not influent either the deme use kafka-client,
jackshirazi commented 2 years ago

Is this still a concern?

jackshirazi commented 2 years ago

Lacking feedback, so closing this now but please reopen if it's still and issue