search

apache / fury

A blazingly fast multi-language serialization framework powered by JIT and zero-copy.
https://fury.apache.org/
Apache License 2.0
3.07k stars 241 forks source link

[Java] In an environment where the ThreadContextClassLoader is not an AppClassLoader, deserialization during asynchronous execution in ForkJoinPool#commonPool may result in a ClassNotFoundException error. #1884

Closed Aliothmoon closed 20 hours ago

Aliothmoon commented 1 week ago

Search before asking

Version

Fury 0.8.0 JDK openjdk version "17.0.4"

Component(s)

Java

Minimal reproduce step

package com.alioth.bootstrap;

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

/**
 * Loading classes from external locations
 * similar to the behavior of Spring Boot's LaunchedURLClassLoader
 *
 */
public class ExternalClassLoader extends ClassLoader {

    /**
     * Classes to be loaded externally
     */
    public static final String[] externalClass = {
            "com.alioth.bootstrap.ExternalClassBootStrap",
            "com.alioth.bootstrap.ExternalClass"
    };

    @Override
    protected Class<?> findClass(String name) throws ClassNotFoundException {
        for (String clz : externalClass) {
            if (clz.equals(name)) {
                Class<?> c = findLoadedClass(name);
                if (c == null) {
                    try {
                        int index = name.lastIndexOf('.');
                        String pos = name.substring(index + 1) + ".class";
                        // prefix System.getProperty("user.dir")
                        Path path = Paths.get(pos);
                        byte[] bytes = Files.readAllBytes(path);
                        return defineClass(name, bytes, 0, bytes.length);
                    } catch (Exception ignore) {
                    }
                }
            }

        }
        return super.findClass(name);
    }
}

/**
 * Classes used for Fury testing
 * Should not appear in the default ClassPath.
 */
public class ExternalClass {
    private int a;
    private String b;

    public int getA() {
        return a;
    }

    public void setA(int a) {
        this.a = a;
    }

    public String getB() {
        return b;
    }

    public void setB(String b) {
        this.b = b;
    }
}

package com.alioth.bootstrap;

import org.apache.fury.Fury;
import org.apache.fury.ThreadLocalFury;
import org.apache.fury.config.Language;

import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.ForkJoinTask;

/**
 * The class used to launch tests  
 * Should not appear in the default ClassPath.
 */
public class ExternalClassBootStrap {

    public static final ClassLoader loader = ExternalClassBootStrap.class.getClassLoader();

    public static final ThreadLocalFury FURY = Fury.builder()
            .withLanguage(Language.JAVA)
            .requireClassRegistration(false)
            .withClassLoader(loader)
            .buildThreadLocalFury();

    static byte[] bytes;

    static {
        ExternalClass obj = new ExternalClass();
        obj.setA(1);
        obj.setB("2");
        bytes = FURY.serialize(obj);
    }

    public static void main(String[] args) throws Exception {

        // The withClassLoader method of FuryBuilder only affects the Fury instance in the current thread, and does not affect Fury instances in other threads.
        // Other Fury instances will, by default, use the ThreadContextClassLoader.
        System.out.println(Thread.currentThread() + " Classloader " + loader);
        Object o1 = FURY.deserialize(bytes);

        ForkJoinTask<?> task = ForkJoinPool.commonPool().submit(() -> {
            ClassLoader cl = Thread.currentThread().getContextClassLoader();
            System.out.println(Thread.currentThread() + " ClassLoader " + cl);
            // ClassNotFoundException occurred because the ThreadContextClassLoader was used instead of the specified loader.
            Object o2 = FURY.deserialize(bytes);
        });

        // Wait for a moment to prevent the ForkJoinPool tasks from being executed by the main thread instead of asynchronously.
        Thread.sleep(500);

        task.get();
    }
}

package com.alioth.bootstrap;

import org.junit.jupiter.api.Test;

import java.lang.reflect.Method;
import java.util.concurrent.ForkJoinPool;

public class FuryTest {

    @Test
    public void testFuryForkJoinPoolAndClassLoader() throws Exception {

        ClassLoader loader = new ExternalClassLoader();
        Thread.currentThread().setContextClassLoader(loader);

        String bootstrap = "com.alioth.bootstrap.ExternalClassBootStrap";
        Class<?> clz = Class.forName(bootstrap, true, loader);

        Method main = clz.getMethod("main", String[].class);
        main.invoke(null, new Object[]{new String[]{}});
    }

}

What did you expect to see?

Complete the deserialization process correctly.

What did you see instead?

An incorrect class loader was selected, making it unable to retrieve the metadata correctly

The stack trace is as follows:

Thread[main,5,main] Classloader com.alioth.bootstrap.ExternalClassLoader@69b2283a
Thread[ForkJoinPool.commonPool-worker-1,5,main] ClassLoader jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b
2024-10-13 12:02:38 INFO  Fury:160 [ForkJoinPool.commonPool-worker-1] - Created new fury org.apache.fury.Fury@40f1c094

java.util.concurrent.ExecutionException: java.lang.IllegalStateException: java.lang.IllegalStateException: Class com.alioth.bootstrap.ExternalClass not found from classloaders [jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b, jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b]

    at java.base/java.util.concurrent.ForkJoinTask.reportExecutionException(ForkJoinTask.java:605)
    at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:981)
    at com.alioth.bootstrap.ExternalClassBootStrap.main(ExternalClassBootStrap.java:45)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at com.alioth.bootstrap.FuryTest.testMultiThreadClassLoader(FuryTest.java:25)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
    at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
    at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
    at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: Class com.alioth.bootstrap.ExternalClass not found from classloaders [jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b, jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b]
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
    at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:562)
    at java.base/java.util.concurrent.ForkJoinTask.reportExecutionException(ForkJoinTask.java:604)
    ... 76 more
Caused by: java.lang.IllegalStateException: Class com.alioth.bootstrap.ExternalClass not found from classloaders [jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b, jdk.internal.loader.ClassLoaders$AppClassLoader@63947c6b]
    at org.apache.fury.resolver.ClassResolver.loadClass(ClassResolver.java:1849)
    at org.apache.fury.resolver.ClassResolver.loadClass(ClassResolver.java:1828)
    at org.apache.fury.resolver.ClassResolver.populateBytesToClassInfo(ClassResolver.java:1765)
    at org.apache.fury.resolver.ClassResolver.loadBytesToClassInfo(ClassResolver.java:1750)
    at org.apache.fury.resolver.ClassResolver.readClassInfoFromBytes(ClassResolver.java:1737)
    at org.apache.fury.resolver.ClassResolver.readClassInfo(ClassResolver.java:1659)
    at org.apache.fury.Fury.readRef(Fury.java:860)
    at org.apache.fury.Fury.deserialize(Fury.java:792)
    at org.apache.fury.Fury.deserialize(Fury.java:714)
    at org.apache.fury.ThreadLocalFury.deserialize(ThreadLocalFury.java:127)
    at com.alioth.bootstrap.ExternalClassBootStrap.lambda$main$0(ExternalClassBootStrap.java:40)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1375)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.lang.ClassNotFoundException: com.alioth.bootstrap.ExternalClass
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:467)
    at org.apache.fury.resolver.ClassResolver.loadClass(ClassResolver.java:1838)
    ... 16 more

Anything Else?

  1. The ForkJoinWorkerThread in ForkJoinPool#commonPool sets the ThreadContextClassLoader in LTS JDK 11 JDK17 JDK21 ，however, this issue does not occur in JDK 8., which can cause the aforementioned issues for some common APIs, such as CompletableFuture when no specific thread pool is provided. 
    ForkJoinWorkerThread(ThreadGroup group, ForkJoinPool pool,
                     boolean useSystemClassLoader, boolean isInnocuous) {
    super(group, null, pool.nextWorkerThreadName(), 0L);
     ...  // Omitting certain content.
    if (useSystemClassLoader) // true
        super.setContextClassLoader(ClassLoader.getSystemClassLoader()); // AppClassLoader
}

    2.Why does the FuryBuilder#withClassLoader method's ClassLoader only apply to the Fury instance in the current thread, and not to all Fury instances under the ThreadSafeFury? Is this due to considerations related to GC?

Are you willing to submit a PR?

chaokunyang commented 1 week ago

It's a bug, ThreadSafeFury should use classloader in FuryBuilder as the default classloader. #1878 also has similar issue.