search

cloudfoundry / java-buildpack

Cloud Foundry buildpack for running Java applications
Apache License 2.0
433 stars 2.59k forks source link

Multiple apps failing to start under `v4.64.0` #1046

Closed coffee-squirrel closed 6 months ago

coffee-squirrel commented 7 months ago

After moving from v4.63.1 to v4.64.0 we found multiple (TBD if all) app deployments started failing with the stack traces shown below.

Common things found so far:

  1. Java 8
  2. JBP_CONFIG_METRIC_WRITER='{enabled: false}'
  3. TAS v4.0.14 (cflinuxfs4)

We've reverted to v4.63.1 and verified subsequent deployments were successful and the instances started cleanly.

I'll try to add more info next week.

2023-12-16 edit: One app had the Auto Reconfiguration-related JARs added due to use of Spring Cloud Connectors (we've been on teams about eliminating this since last year). Disabling that (JBP_CONFIG_SPRING_AUTO_RECONFIGURATION='{enabled: false}') appears to have cleared up the issue under 4.64.0 for this app, but the owning team probably has some work to do. I believe another app that was impacted doesn't use Auto Reconfiguration, Spring Cloud Connectors, or java-cfenv, though, so I'll look into that next.

2023-12-15 14:15:55.030 ERROR [myapp,,,] 7 --- [           main] o.s.boot.SpringApplication               : Application startup failed
"java.lang.IllegalStateException: Error processing condition on org.springframework.boot.actuate.autoconfigure.AuditAutoConfiguration$AuditEventRepositoryConfiguration
    at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)
    at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:102)
    at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$TrackedConditionEvaluator.shouldSkip(ConfigurationClassBeanDefinitionReader.java:444)
    at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:127)
    at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:116)
    at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:320)
    at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:228)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:272)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:92)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:524)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:123)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:666)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:353)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:300)
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:135)
    at com.myorg.myapp.App.main(App.java:15)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
    at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:59)
Caused by: java.lang.reflect.MalformedParameterizedTypeException: null
    at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.validateConstructorArguments(ParameterizedTypeImpl.java:58)
    at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:51)
    at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:92)
    at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105)
    at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140)
    at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
    at sun.reflect.generics.repository.MethodRepository.getReturnType(MethodRepository.java:68)
    at java.lang.reflect.Method.getGenericReturnType(Method.java:255)
    at org.springframework.core.MethodParameter.getGenericParameterType(MethodParameter.java:397)
    at org.springframework.core.SerializableTypeWrapper$MethodParameterTypeProvider.getType(SerializableTypeWrapper.java:338)
    at org.springframework.core.SerializableTypeWrapper.forTypeProvider(SerializableTypeWrapper.java:146)
    at org.springframework.core.ResolvableType.forType(ResolvableType.java:1346)
    at org.springframework.core.ResolvableType.forMethodParameter(ResolvableType.java:1249)
    at org.springframework.core.ResolvableType.forMethodParameter(ResolvableType.java:1217)
    at org.springframework.core.ResolvableType.forMethodReturnType(ResolvableType.java:1160)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:759)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:638)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:607)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1496)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1018)
    at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanTypeForNonAliasDefinition(BeanTypeRegistry.java:217)
    at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanTypeForNonAliasDefinition(BeanTypeRegistry.java:195)
    at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanType(BeanTypeRegistry.java:188)
    at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.updateTypesIfNecessary(BeanTypeRegistry.java:175)
    at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.getNamesForType(BeanTypeRegistry.java:112)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:171)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:158)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchingBeans(OnBeanCondition.java:128)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:103)
    at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
    ... 24 common frames omitted

"at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:102)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$TrackedConditionEvaluator.shouldSkip(ConfigurationClassBeanDefinitionReader.java:444)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:127)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:116)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:320)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:228)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:272)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:92)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:524)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:123)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:666)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:353)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:300)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:135)
at com.myorg.myapp.App.main(App.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:59)

2023-12-15 14:15:55.033  WARN [myapp,,,] 7 --- [           main] o.s.boot.SpringApplication               : Unable to close ApplicationContext
"java.lang.reflect.MalformedParameterizedTypeException: null
    at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.validateConstructorArguments(ParameterizedTypeImpl.java:58)
    at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:51)
    at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:92)
    at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105)
    at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140)
    at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
    at sun.reflect.generics.repository.MethodRepository.getReturnType(MethodRepository.java:68)
    at java.lang.reflect.Method.getGenericReturnType(Method.java:255)
    at org.springframework.core.MethodParameter.getGenericParameterType(MethodParameter.java:397)
    at org.springframework.core.SerializableTypeWrapper$MethodParameterTypeProvider.getType(SerializableTypeWrapper.java:338)
    at org.springframework.core.SerializableTypeWrapper.forTypeProvider(SerializableTypeWrapper.java:146)
    at org.springframework.core.ResolvableType.forType(ResolvableType.java:1346)
    at org.springframework.core.ResolvableType.forMethodParameter(ResolvableType.java:1249)
    at org.springframework.core.ResolvableType.forMethodParameter(ResolvableType.java:1217)
    at org.springframework.core.ResolvableType.forMethodReturnType(ResolvableType.java:1160)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:759)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:638)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:607)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1496)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:425)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:395)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:515)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:508)
    at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1190)
    at org.springframework.boot.SpringApplication.getExitCodeFromMappedException(SpringApplication.java:785)
    at org.springframework.boot.SpringApplication.getExitCodeFromException(SpringApplication.java:773)
    at org.springframework.boot.SpringApplication.handleExitCode(SpringApplication.java:760)
    at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:715)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:135)
    at com.myorg.myapp.App.main(App.java:15)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
    at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:59)

"at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.validateConstructorArguments(ParameterizedTypeImpl.java:58)
at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:51)
at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:92)
at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105)
at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140)
at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at sun.reflect.generics.repository.MethodRepository.getReturnType(MethodRepository.java:68)
at java.lang.reflect.Method.getGenericReturnType(Method.java:255)
at org.springframework.core.MethodParameter.getGenericParameterType(MethodParameter.java:397)
at org.springframework.core.SerializableTypeWrapper$MethodParameterTypeProvider.getType(SerializableTypeWrapper.java:338)
at org.springframework.core.SerializableTypeWrapper.forTypeProvider(SerializableTypeWrapper.java:146)
at org.springframework.core.ResolvableType.forType(ResolvableType.java:1346)
at org.springframework.core.ResolvableType.forMethodParameter(ResolvableType.java:1249)
at org.springframework.core.ResolvableType.forMethodParameter(ResolvableType.java:1217)
at org.springframework.core.ResolvableType.forMethodReturnType(ResolvableType.java:1160)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:759)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:638)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:607)
at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1496)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:425)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:395)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:515)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:508)
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1190)
at org.springframework.boot.SpringApplication.getExitCodeFromMappedException(SpringApplication.java:785)
at org.springframework.boot.SpringApplication.getExitCodeFromException(SpringApplication.java:773)
at org.springframework.boot.SpringApplication.handleExitCode(SpringApplication.java:760)
at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:715)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:135)
at com.myorg.myapp.App.main(App.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:59)
thelangley commented 7 months ago

Same. Our customers are experience similar errors. Experiencing the same even with JBP_CONFIG_SPRING_AUTO_RECONFIGURATION='{enabled: false}' set

They're currently checking for conflicting dependencies in their poms. Apps push successfully when using v4.63.1

pivotal-david-osullivan commented 7 months ago

v4.64.0 has upgraded the version of the Client Certificate Mapper library to 2.0.0 with the aim of supporting Boot 3.x and the change to jakarta.* packages as well as maintaining support for 2.x and the javax.* naming.

It's possible a bug has been been introduced - can you please share more details about the affected apps while we investigate? e.g.

  1. Are all apps used with this buildpack affected on the platform, or a subset?
  2. If a subset, do they share common config? Only Java 8, Boot dependency versions etc.
  3. Sharing the pom.xml if possible may be helpful also
thelangley commented 7 months ago

Looks spring-boot-starter-parent related. Apps with 1.5.2.RELEASE or 1.5.9.RELEASE have encountered issues. I know, soooo old. Apps with 2.0.x appear to work ok.

Java 8 only so far

coffee-squirrel commented 7 months ago

Yep, looks like our apps that were impacted were on 1.5.12.RELEASE or 1.5.22.RELEASE. Other apps on 2.x deployed successfully.

thelangley commented 7 months ago

Looks like a new version of the client-certificate-mapper has been cut which should re-enable customers to use 1.5.x again.

https://github.com/cloudfoundry/java-buildpack-client-certificate-mapper/commits/v2.0.1/

Are there any plans for a teeny/minor java-buildpack release to incorporate this? It'll get our customers over this hump then early in the new year, they can complete the fun job of updating their dependencies

pivotal-david-osullivan commented 7 months ago

@coffee-squirrel @thelangley We plan to release a new version this week, however we strongly recommend users update to at least the latest supported Spring Boot 2.x, as commercial support for 1.5 ended in 2020 and the Java Buildpack cannot guarantee compatibility with such outdated configurations.

thelangley commented 7 months ago

Totally behind that. We're pushing them to do the right thing and update to a more modern version, maybe even one from this decade.

Hopefully this'll focus minds on why you should care about your dependencies :)

coffee-squirrel commented 7 months ago

Yep, completely understand and appreciate you all putting in any time on this. We'll use this to keep trying to get teams to update their stuff.

After the new version hits PivNet I'll get some apps redeployed and post back with results.

coffee-squirrel commented 6 months ago

We've used v4.65.0 to successfully deploy the apps that were having issues under v4.64.0. Thanks again!