Region Not Found with Spring Boot due to Java 11 ForkJoinPool `ClassLoader` change

[gidxl03]

Problem I found that after upgrading from Java 8 to Java 11, my Spring boot app intermittently (80% of the time) fails to start due to

Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'xxxRepository' defined in com.xxx.gemfire.repository.CommandRepository defined in @EnableGemfireRepositories declared on XXXXConfiguration: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Region [...] for Domain Type [...] using Repository [...] was not found; You must configure a Region with name [....] in the application context"

Debuging ... So the Repository classes are found but the adjacent Region classes are not. Debuging soon revealed the issue was caused by @Region beans not being found by Spring's ClassPathScanningCandidateComponentProvider. With trace debugging, I found that when the problem occurs 1) the classloader is of type 'jdk.internal.loader.ClassLoaders' and 2)the thread of type 'ForkJoinPool.commonPool-worker'

Root Cause From there I found https://github.com/spring-projects/spring-boot/issues/6626 and https://stackoverflow.com/questions/49113207/completablefuture-forkjoinpool-set-class-loader which advise that

In Java SE 9, threads that are part of the fork/join common pool will always return the system class loader as their thread context class loader. In previous releases, the thread context class loader may have been inherited from whatever thread causes the creation of the fork/join common pool thread, ...

My understanding is that because SpringBoot puts classes in folder BOOT_INF, the Spring classloader is needed to find them.

Line 212 of https://github.com/spring-projects/spring-data-gemfire/blob/main/src/main/java/org/springframework/data/gemfire/config/annotation/support/GemFireComponentClassTypeScanner.java requests that all classes be streamed and loaded in parallel and so the ForkJoinPool will attempt and fail to find the Region classes in the SpringBoot jar. stream(this.spliterator(), true)

Possible fixes

1. In the same spring-data-geode class, add new line componentProvider.setResourceLoader(new DefaultResourceLoader(getEntityClassLoader())); before the return in method newClassPathScanningCandidateComponentProvider
2. Change line 212 to stream(this.spliterator(), false)
3. Define a custom ForkJoinWorkerThreadFactory() with Spring Security Context as per the stackoverflow link above.

Notes In the cases where the app starts, ForkJoinPool is not used at all and the JVM/OS decides to execute the entire stream sequentially using the main thread. I never found that only some of the Regions where loaded.

See also https://stackoverflow.com/questions/72740543/issue-with-spring-boot-gemfire-integration which also describes a problem loading Region beans

[jxblum]

Thank you for reporting this issue.

NOTE: Spring Data for Apache Geode (SDG) is nearing its OSS EOL, as of November 2023 (see here for complete details). SDG, and by extension, Spring Boot for Apache Geode, will no longer receive updates without purchasing commercial support (through a VMware Spring Runtime license). As such, you are encouraged to replace Spring Data for Apache Geode with Spring Data for VMware GemFire where, and as soon as possible.

Having noted the above statement, I am going to make 2 changes to SDG.

1) First, I am going to add componentProvider.setResourceLoader(new DefaultResourceLoader(getEntityClassLoader()));

2) Then, I am going to make the parallelization of the stream(..) used for classpath component scanning configurable (using a new, hidden Spring Data Geode property, "spring.data.gemfire.classpath.scan.parallel"), defaulting to false.

With 2, users can set this property in Spring Boot's application.properties since the value is resolved from the Spring Environment.

Given the core Spring Framework, Spring Container's (prior) bean indexing capabilities, and now comprehensive AOT (eventual CrAC and Projet Leydon support) it is not strictly necessary to parallelize the classpath component scan. In fact, by default, it should not.

Still, if users need this capability, then it is a simple matter to configure this behavior using the new property from #2 above.

[jxblum]

Fortunately, this fix will go out tomorrow, during our regularly scheduled Spring Data releases.

[gidxl03]

Many thanks John for the rapid turnaround, prolific as ever!

EOL notice for SDG is noted.

I've benefited from your many detailed stackoverflow explanations over the years and want to say thanks for that too :)

[jxblum]

Thank you for the kind feedback. Wishing you all the best.