Devtools: Same class with different classloader causing NoSuchBeanDefinitionException

[cemo]

@philwebb I have started to check our applications and came across an issue. (Sorry if It is already fixed)

Short description: My applications can not find some of beans when application starts.

Detailed description: I have debugged and found that If a class is loaded with two different class loader, the java.lang.Class#isAssignableFrom seems can not handle correctly. This is causing a problem in org.springframework.util.ClassUtils#isAssignable which is used for bean comparison. As a result a bean not found exception is raised.

I have checked each class and noticed that classes are loaded by AppClassLoader and RestartClassLoader.

This bean is registered by @Import configuration class. Spring Framework is registering beans with AppClassLoader but classes of other beans are loaded by RestartClassLoader.

[cemo]

I have also tried with 1.3.0.BUILD-SNAPSHOT which is also causing an exception.

[philwebb]

Do you have a sample application we could try? Usually Spring will use the context classloader to load beans (which should be the RestartClassLoader but we have seen some problems when Class.forName is used.

[cemo]

Sorry @philwebb I was on vacation. I am trying to understand how this whole stuff is working. I will let you know details.

[sergiorc]

I have found the same issue using:

spring-boot 1.3.0-BUILD-SNAPSHOT spring-dev-tools 1.3.0-BUILD-SNAPSHOT spring-hateoas 0.18.8-BUILD-SNAPSHOT

To reproduce the issue, spring-hateoas project MUST be open in your IDE workspace (eclipse in my case).

Spring HATEOAS HypermediaSupportBeanDefinitionRegistrar class is creating a DelegatingRelProvider with "_relProvider" name:

        BeanDefinitionBuilder delegateBuilder = BeanDefinitionBuilder.rootBeanDefinition(DelegatingRelProvider.class);
        delegateBuilder.addConstructorArgValue(registryBeanDefinition);

        AbstractBeanDefinition beanDefinition = delegateBuilder.getBeanDefinition();
        beanDefinition.setPrimary(true);
        registry.registerBeanDefinition(DELEGATING_REL_PROVIDER_BEAN_NAME, beanDefinition);

But it doesn't qualify as a org.springframework.hateoas.RelProvider instance in org.springframework.boot.autoconfigure.hateoas.HypermediaAutoConfiguration$HypermediaConfiguration$HalObjectMapperConfiguration:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.hateoas.HypermediaAutoConfiguration$HypermediaConfiguration$HalObjectMapperConfiguration': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.hateoas.RelProvider org.springframework.boot.autoconfigure.hateoas.HypermediaAutoConfiguration$HypermediaConfiguration$HalObjectMapperConfiguration.relProvider; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type [org.springframework.hateoas.RelProvider] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true), @org.springframework.beans.factory.annotation.Qualifier(value=_relProvider)}
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:334)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)

Going a bit deeper, I've found the same case that @cemo. When testing if DelegatingRelProvider.class is assignable to RelProvider.class (org.springframework.util.ClassUtils.isAssignable) the return is false because classloaders are distinct.

DelegatingRelProvider classloader is RestartClassLoader. RelProvider classloader is AppClassLoader.

Closing spring-hateoas in the IDE, or removing spring-dev-tools from classpath prevents any error. I suppose the issue can be reproduced with any other project instead spring-hateoas.

[philwebb]

@sergiorc I've hit the same issue when developing devtools. It's to do with the way that the split classloader is created. Since eclipse will be exposing spring-hateos as an exploded application devtools thinks it is a candidate for monitoring.

We have some logic in ChangeableUrls that detects some Spring Jars, we could potentially extend this but it's a bit tricky to know if a jar is a Spring library or not (only going on the name).

The easiest workaround for now is to disable inter-project resolution in eclipse or simply work on spring-hatoes in a different workspace.

[philwebb]

@cemo are you still having the issue? Did you have other Spring projects open in your workspace?

[cemo]

I have created a sample application with our codebase with not all of our modules. This application was working properly but when I started run at production application, I am having same issue. I hope that I will give a try with latest beta and narrow issue.

[sergiorc]

@philwebb But, this issue will only happen with spring libraries? I supposed the problem is comparing any class between both classloaders (AppClassLoader and RestartClassloader).

[snicoll]

It will happen indeed with any library that deserialize content. Caching libraries, in particular, are affected.

[sergiorc]

Ok, thanks by the info

[jceloi]

Just to confirm it, I've experienced the same problem with spring-security oauth2 jdbc store, which serializes objects.

For instance when inspecting the classloaders of my object and the class as used within the application I get those two :

• sun.misc.Launcher$AppClassLoader@15316f2 <- deserialized
• org.springframework.boot.devtools.restart.classloader.RestartClassLoader@c8de43 <- currently executing code

Hence the classcast with my class not being able to be cast to itself.

[cemo]

@philwebb @wilkinsona

I really would like to help you. But I have limited knowledge on this area. I have no idea why some of my classes are loaded by RestartClassLoader and AppClassLoader.

But I have noticed that @EnableXXX style bean loading is causing issue on our side. We have a huge code base and some beans are loaded by "@EnableXXX" style configurations and some of them are by AutoConfigurations. EnableXXX style beans are loaded AppClassLoader and others by RestartClassLoader and this is causing issue.

Please let me know for further helps.

[wilkinsona]

@cemo Thanks. Every class that's available directly on the filesystem, i.e. not packaged in a jar, should be loaded by the RestartClassLoader. Everything that's in a jar should be loaded by the AppClassLoader (which is RestartClassLoader's parent).

The different behaviour that you've observed for beans loaded via @EnableXXX is interesting. Are you just @Importing another configuration class in your @EnableXXX annotation, or doing something more sophisticated with an ImportSelector?

[cemo]

I am using such a pattern:

@Import(EnableXXX.XXXConfig.class)
public @interface EnableXXX {

   @Configuration
   static class XXXConfig {

      @Bean
      public MyProcessService myProcessService(MyFactory myFactory) {
        // removed
      }
   }
}

I have some additional observations:

1. When I set a breakpoint on java.lang.ClassLoader#loadClass(java.lang.String, boolean) I can see that many of my classes at filesystem are loaded by AppClassLoader. This is causing an issue by chaining interestingly.
2. When I have an autoconfiguration like this:

@Configuration
@Import(EnableXXX.XXXConfig.class)
public class XXXAutoConfiguration {
}

Problem is still persist. However when I do not import and instead put whole bean logic inside XXXAutoConfiguration as this:

@Configuration
public class XXXAutoConfiguration {

     @Bean
      public MyProcessService myProcessService(MyFactory myFactory) {
        // removed
      }
}

Problem is solved. I am still trying to reproduce with a simple application.

[wilkinsona]

When I set a breakpoint on java.lang.ClassLoader#loadClass(java.lang.String, boolean) I can see that many of my classes at filesystem are loaded by AppClassLoader

I suspect this is the root of the problem. A breakpoint on ChangeableUrls.isReloadable(URL) might help to show what's going on. For any URLs pointing to classes on the filesystem, isFolderUrl should return true.

[wilkinsona]

@cemo Another thought: the stack trace when you can see an application class being loaded byAppClassLoader would be very useful. Assuming that those classes have been correctly identified as reloadable, that would tell us that whatever's loading the class is just using the wrong class loader.

[cemo]

Your comments make sense since I have loaded EnableXXX classes by jar.

Now I am trying to run a scenario where some of my configuration inside a library class. I will try to investigate such a scenario:

Project A: Has a ServiceA Library B (jar file) : ConfigurationB needs ServiceA

I will let you know.

[cemo]

I have just reproduced issue :) In 5 minutes I will upload.

[cemo]

In order to reproduce:

I have 3 project:

• Library A
• Library B depends on Library A
• Demo project depends on Library B and A

---

In order to reproduce:

1. Please install all libraries at command line in particular order: Library A, Library B, Demo
2. Import Library A and Demo project into your IDE but not Library B. (I am using IDEA)
3. Put a conditional break point inside this method org.springframework.util.ClassUtils#isAssignable like this:

lhsType.getSimpleName().equals(rhsType.getSimpleName()) && !lhsType.getClassLoader().equals(rhsType.getClassLoader())

You will see that there is same class ServiceA in both AppClassLoader and RestartClassLoader. And equality check is returning wrong. This is preventing injection and thus No qualifying bean of type [org.a.ServiceA] is thrown.

Here is the project link: https://www.dropbox.com/s/jiduhaz6qj3hgtd/demo%202.zip?dl=0

[wilkinsona]

Thank you. I am 99% certain that this is a variant of https://github.com/spring-projects/spring-boot/issues/3805. The problem is that Library B, as it's in a JAR, is loaded by the app class loader. This means that any application classes that it loads, i.e. those that should be loaded by the restart class loader are loaded by the app class loader instead. We have a fix in mind for #3805 that @philwebb has prototyped. Based on the discussion I had with him this morning, I'm hopeful that it'll fix this issue too.

[philwebb]

I agree with Andy that this is a variant of #3805 but I don't think that the fix for #3805 will fix it. Devtools works by creating a split classloader, the idea being that the application classes are in a loader that is thrown away and library classes are in the one that's kept. Usually this works fine because library classes (like Spring/Jackson etc) have no direct dependencies on your user defined classes.

The problem in your example is that "B" has a dependency on "A" but has ended up in the lower classloader because it's not unpacked:

+----------------+
|    A + Demo    | (restart classloader)
+----------------+   |
                     | can use class in
+----------------+   v
| B + Other Libs | (application classloader)
+----------------+

What we need to do is find a way to pull 'B' up into the restart classloader. Perhaps we could do this with some system property, or perhaps we could try to do it automatically perhaps based on package names.

[wilkinsona]

Ah, crap. It's essentially the same problem as Orika has (#3697).

[cemo]

@philwebb What do you think about putting a file inside each necessary jar by either maven or gradle plugins to support reloading by restart classloader?

[philwebb]

@cemo It's tricky because that only works if you are responsible for generating those JARs. In the Orika case, it's someone else's JAR.

Is there any specific reason why in your case Library B can't be imported into your IDE?

[cemo]

Actually we have dozens Library B's which are infrastructure codes. Our codebase can be considered into two parts. A highly reusable components of infrastructure codes and our websites which are available for end users. Our websites are maintained by junior developers and I do not want to confuse their minds. Their primary goals are using libraries, many of them are working in a declarative manner, in an efficient way. Our infrastructure codes are versioned whereas websites are working less restrictive way. Importing all infrastructure codes requires checking out necessary git tags etc... This process is not easy and require additional steps even in github to authenticate users to pull necessary projects. In contrast to this process, current situation is quite lightweight. They just need to declare a dependency and all the magic happens thanks to you and our glue codes.

Another solution might be changing logic inside bean comparison in Spring Core. The root cause is actually having same class with different classloaders. I am not sure how this idea sounds but have you ever considered to change in Spring Core to check equality by only their fully qualified name? I am currently not aware of implications of this decision but just make you sure that you have considered it.

[philwebb]

I've added support for META-INF/spring-devtools.properties files which can be used to pull jars up to the restart classloader. Hopefully you can add a restart.include.... regex to solve your issue.

[cemo]

I can confirm that this issue is fixed. Thanks for your efforts.

PS: Don't forget removing in progress tag. :)

[gaeloberson]

I'm having trouble to setup META-INF/spring-devtools.properties in my maven project. Could you please provide me an advice? I put the file under <project-root>/src/main/resources/META-INF but apparently it does not get read by devtools. the regexp I use is restart.include.droolslibs=/drools-[\\s\\S]+\.jar but the drools Classes are still loaded by the AppClassLoader.

[philwebb]

@gaeloberson That should be the right place. Try putting a breakpoint on DevToolsSettings.isRestartInclude(...) to see if the restartIncludePatterns get loaded and if the regex applies cleanly. If you don't get anywhere please open a new issue (ideal with a sample project to reproduce the problem).

[yantantether]

For anyone hitting this issue with Drools, I found this config worked for me:

META-INF/spring-devtools.properties

restart.include.drools=/drools-[\\s\\S]+\.jar
restart.include.kie=/kie-[\\s\\S]+\.jar

[Drezir]

I have just discovered similar problem with isAssignableFrom while developing project with apache kafka + avro.

I have one @KafkaListener

    @KafkaListener(topics = "${x1.notification.topic.contract-proposal}")
    @Transactional("transactionManager")
    public void listenForContractProposal(Udalost event) {

I created REST API to send HTTP body to kafka using same avro object Udalost

    @PostMapping("/knz_smlouva")
    public TestingResponse knzSmlouva(@RequestBody Udalost udalost) {
        kafkaTemplate.send(knzSmlouvaTopic, udalost).join();

Message is produced to kafka but there is a problem in line PayloadMethodArgumentResolver#139

Class<?> targetClass = resolveTargetClass(parameter, message);
Class<?> payloadClass = payload.getClass();
if (!ClassUtils.isAssignable(targetClass, payloadClass)) { // here it does not match

Target and payload class are same Udalost.class but with different unnamed module I guess and that causes this condition to fail and on next lines it tries to convert Udalost.class -> Udalost.class using conversion service and it fails.

If I remove devtools, problem is gone.

[wilkinsona]

@Drezir please open a new issue with a minimal sample that reproduces the problem