Incorrect classloader used by common ForkJoinPool when using Executable Jar

[pcimcioch]

Context

I created this issue as a bug report or enhancement proposal - depending on how would you classify current behaviour.

I have a spring application that I am building using "org.springframework.boot gradle" plugin. This plugin builds Executable Jar and War as described in documentation: https://docs.spring.io/spring-boot/docs/current/reference/html/executable-jar.html

Problem

Executable Jar uses custom class loader: org.springframework.boot.loader.launch.LaunchedClassLoader when running the application. This class loader is not propagated to the common ForkJoinPool, which uses system class loader by default.

Take a code like that:

IntStream.rangeClosed(0, 4)
    .parallel()
    .forEach(i -> System.out.println(Thread.currentThread().getName() + " " + Thread.currentThread().getContextClassLoader()));

It will produce following output:

ForkJoinPool.commonPool-worker-1 jdk.internal.loader.ClassLoaders$AppClassLoader@33909752
ForkJoinPool.commonPool-worker-2 jdk.internal.loader.ClassLoaders$AppClassLoader@33909752
ForkJoinPool.commonPool-worker-1 jdk.internal.loader.ClassLoaders$AppClassLoader@33909752
http-nio-8080-exec-1 TomcatEmbeddedWebappClassLoader
  context: ROOT
  delegate: true
----------> Parent Classloader:
org.springframework.boot.loader.launch.LaunchedClassLoader@1a6c5a9e

We have 4 tasks to execute in parallel. For such execution, java uses commom ForkJoinPool.

One of the tasks executed on current thread (http-nio-8080-exec-1) and it sees "correct" class loader: LaunchedClassLoader. Other three tasks executed on separate threads, that see "incorrect", system class loader: AppClassLoader

This causes issues if we try to execute in parallel piece of code that needs to have access to the "proper" class loader.

This behaviour is even described in the documentation: https://docs.spring.io/spring-boot/docs/current/reference/html/executable-jar.html#appendix.executable-jar.restrictions

System classLoader: Launched applications should use Thread.getContextClassLoader() when loading classes (most libraries and frameworks do so by default). Trying to load nested jar classes with ClassLoader.getSystemClassLoader() fails.

The problem is that the the class that we are considering here - common ForkJoinPool - is a big part of JDK itself

Possible Fix / Enhancement

Common ForkJoinPool can be configured to use different ThreadFactory (by setting java.util.concurrent.ForkJoinPool.common.threadFactory system property) - for example, custom ThreadFactory that returns threads with LaunchedClassLoader

Workarounds

Configure custom thread factory

You can create your custom thread factory like so:

public class MyForkJoinWorkerThreadFactory implements ForkJoinWorkerThreadFactory {
    @Override
    public final ForkJoinWorkerThread newThread(ForkJoinPool pool) {
        return new MyForkJoinWorkerThread(pool);
    }

    private static class MyForkJoinWorkerThread extends ForkJoinWorkerThread {
        private MyForkJoinWorkerThread(final ForkJoinPool pool) {
            super(pool);
            setContextClassLoader(Thread.currentThread().getContextClassLoader());
        }
    }
} 

You can set up system property java.util.concurrent.ForkJoinPool.common.threadFactory=foo.bar.MyForkJoinWorkerThreadFactory to make common ForkJoinPool use this thread factory.

Problem: ForkJoinPool will use system class loader to find foo.bar.MyForkJoinWorkerThreadFactory, so it must be part of spring boot launcher class path

Don't use common ForkJoinPool

We could use custom ForkJoinPool with custom ForkJoinWorkerThreadFactory like

try(ForkJoinPool pool = new ForkJoinPool(4, new MyForkJoinWorkerThreadFactory(), null, false)) {
    pool.submit(() -> IntStream.rangeClosed(0, 4).parallel()
        .forEach(i -> System.out.println(Thread.currentThread().getName() + " " + Thread.currentThread().getContextClassLoader())););
}

Problems:

• It's very verbose, you need to wrap all you application entry points in custom ForkJoinPool
• Common ForkJoinPool implementation is a bit different then ForkJoinPool. Namely, it always uses current thread as one of the worker threads. This functionality is very useful in some contexts, and you can't achieve it using custom ForkJoinPool

Don't use Executable Jar format

You can try building jar for you spring application withou using Executable Jar (without the launcher). Documentation even lists some alternative methods in Alternative Single Jar Solutions

Problem: Building fat jar for spring application is quite hard. I tried using Gradle Shadow Plugin, but it is hard to correctly merge every necessery file. I didn't found any (up to date) solution that would worker

Unpack fat jar

You can also unpack fat jar created by spring boot plugin and run your application manually, without the launcher (as described in https://stackoverflow.com/questions/58746223/are-there-caveats-to-not-using-the-spring-boot-classloader-in-production)

$ jar -xf myapp.jar
$ java -cp "BOOT-INF/classes:BOOT-INF/lib/*" com.example.MyApplication

Problem: Won't work in environments where you have to provide executable jar file

[philwebb]

See #19427 for some previous discussion on this topic.

[philwebb]

We're going to investigate if we can provide our own ForkJoinWorkerThreadFactory and possibly configure the system property by default. If we do this, we'll also need a way to opt-out of our version.

[Sasivarnan1988]

I am having the same issue even while using the completable future without executors. I tried custom thread factory and set that through environment variable but still I could see jdk loader incorrect one. Any help? This is on Jdk17

[wilkinsona]

@Sasivarnan1988 A correctly configured thread factory should resolve the problem. If that isn't working for you, please follow up on Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

[pcimcioch]

@Sasivarnan1988 The problem with custom thread factory is that java uses "system classloader" to instantiate it. And your application classes are not availabe in system class loader. It may be hard to notice, as it will fail silently. You would have to somehow pack your custom thread factory together with the launcher code in the jar, which I'm not sure is possible using gradle/maven spring boot plugin

[wilkinsona]

It's reasonably straightforward with Gradle but much harder with Maven. See https://github.com/spring-projects/spring-boot/issues/6626.

[Sasivarnan1988]

Thank you all for the comments. I will post it in the stack overflow as I have few more doubts on this.

[appleseedexm]

I hope this is not too much off topic, but we're suffering from (very likely) the same issue. A possible fix is to provide a spring-managed ForkJoinPool, which is annoying and not what we want, but it would fix the classloader. We also reproduced that in certain circumstances (more below) the exact same executable jar runs async code (using ForkJoinPool) in a thread with Spring's classloader, and in others with jdk's AppClassLoader (the later fails).

What we also noticed and I wanted some feedback on, this only happens when running more than 2 cores? The above described classloader differences only exist when running with 3+ cores, and in all environments with less cores it uses Spring's classloader. We tested this on the same machines, and on multiple containers and vms. The only alternative to me would be a coincidence that less cores = less fast startup, which leads to some weird jvm setup. Does someone encounter the same behavior / know if this is expected to happen?