BeanCreationNotAllowedException on shutdown

[AndreKoepke]

On a graceful shutdown of Spring, Spring-Cloud-Stream will trigger a BeanCreationNotAllowedException.

BeanCreationNotAllowedException

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'solace-1551031822.BINDER_NAME.errors.bridge55409437': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1237) ~[spring-context-6.1.4.jar:6.1.4]
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.destroyErrorInfrastructure(AbstractMessageChannelBinder.java:909) ~[spring-cloud-stream-4.1.0.jar:4.1.0]
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder$1.afterUnbind(AbstractMessageChannelBinder.java:383) ~[spring-cloud-stream-4.1.0.jar:4.1.0]
    at org.springframework.cloud.stream.binder.DefaultBinding.unbind(DefaultBinding.java:187) ~[spring-cloud-stream-4.1.0.jar:4.1.0]
    at org.springframework.cloud.stream.binding.BindingService.unbindProducers(BindingService.java:405) ~[spring-cloud-stream-4.1.0.jar:4.1.0]
    at java.base/java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:729) ~[na:na]
    at org.springframework.cloud.stream.function.StreamBridge.destroy(StreamBridge.java:311) ~[spring-cloud-stream-4.1.0.jar:4.1.0]
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:211) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1202) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1195) ~[spring-beans-6.1.4.jar:6.1.4]
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1183) ~[spring-context-6.1.4.jar:6.1.4]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1144) ~[spring-context-6.1.4.jar:6.1.4]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174) ~[spring-boot-3.2.3.jar:3.2.3]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1090) ~[spring-context-6.1.4.jar:6.1.4]
    at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145) ~[spring-boot-3.2.3.jar:3.2.3]
    at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
    at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114) ~[spring-boot-3.2.3.jar:3.2.3]
    at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

To reproduce this:

1. Create a Spring application with a graceful shutdown (set server.shutdown to graceful)
2. Create a binding to a topic with an error-channel
3. Do a graceful shutdown
4. Send messages while shutdown
5. Wait for spring.lifecycle.timeout-per-shutdown-phase (default is 30s)

It is not allowed by Spring to get beans when it is in destruction mode. Maybe this code should be moved to @PreDestruction phase of the error-channel-beans. https://github.com/spring-cloud/spring-cloud-stream/blob/7845be50effe311891a72243556cd3d11cad7cc9/core/spring-cloud-stream/src/main/java/org/springframework/cloud/stream/binder/AbstractMessageChannelBinder.java#L909-L912

Same issue as #1870, #2171 and #2193 (all are already closed because of duplicate or too old)

[olegz]

I can't reproduce it. Perhaps I a not doing exactly what you have, so consider providing a small reproducible sample. You can attach it a ZIP file or push it to the GitHub repo and leave a link

[AndreKoepke]

I'm getting it almost always. But I have a lot of consumers. Maybe it is a race-condition?

I will make a better example - perhaps it will be next week.

[olegz]

Closing due to lack of follow up from the reporter

[AndreKoepke]

Sorry, I forgot to answer.

But now, I was able to reproduce it. I'm not sure who in the chain doing something wrong - maybe it can be stay closed. It is your choice @olegz. :)

This bug happen when someone try to send a message via Solace when Spring is going to shutdown. Here is an example:

/*
 *  Place this configuration in a Spring-Application. Set to destination to an existing binder and ensure that the shutdown
 *  process takes a couple of seconds.
 */
@Configuration
public class CauseABugConfig {

    private final StreamBridge streamBridge;

    public CauseABugConfig(StreamBridge streamBridge) {
        this.streamBridge = streamBridge;
    }

    @PostConstruct
    public void sendMessage() {
        var timerTask = new TimerTask() {
            @Override
            public void run() {
                streamBridge.send("umplalumpa", "hi");
            }
        };
        new Timer().schedule(timerTask, 1000, 1000);
    }

Possible solutions are:

1. Ensure that Solace-Consumers cannot be shutdown before StreamBridge (@DependsOn()?).
2. Error in Layer 8 aka Users of this library should ensure not to send messages on shutdown.

[jesty]

It's happening the same to us in a project using RabbitMQ. I agree that the problem it is when we are closing StreamBridge.

[dima-bzz]

After fix error-handler-definition, an error appears when shutdown app

Error creating bean with name 'kafka-999804794.event-out-0.errors': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

[olegz]

@dima-bzz any stack trace? any way to reproduce it?

[dima-bzz]

@olegz https://github.com/dima-bzz/testkafka/tree/main

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'kafka-1313602972.source1-out-0.errors': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1249) ~[spring-context-6.1.13.jar:6.1.13]
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.destroyErrorInfrastructure(AbstractMessageChannelBinder.java:923) ~[classes/:4.1.3]
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder$1.afterUnbind(AbstractMessageChannelBinder.java:388) ~[classes/:4.1.3]
    at org.springframework.cloud.stream.binder.DefaultBinding.unbind(DefaultBinding.java:186) ~[spring-cloud-stream-4.1.3.jar:4.1.3]
    at org.springframework.cloud.stream.binding.BindingService.unbindProducers(BindingService.java:405) ~[spring-cloud-stream-4.1.3.jar:4.1.3]
    at java.base/java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:589) ~[na:na]
    at org.springframework.cloud.stream.function.StreamBridge.destroy(StreamBridge.java:348) ~[spring-cloud-stream-4.1.3.jar:4.1.3]
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:211) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1202) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1195) ~[spring-beans-6.1.13.jar:6.1.13]
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1195) ~[spring-context-6.1.13.jar:6.1.13]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1156) ~[spring-context-6.1.13.jar:6.1.13]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174) ~[spring-boot-3.2.10.jar:3.2.10]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1102) ~[spring-context-6.1.13.jar:6.1.13]
    at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145) ~[spring-boot-3.2.10.jar:3.2.10]
    at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
    at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114) ~[spring-boot-3.2.10.jar:3.2.10]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

[olegz]

@AndreKoepke I am going to close it, but. . . Since you have mentioned "This bug happen when someone try to send a message via Solace when Spring is going to shutdown", i would contact Solace team to see if they have something to add since this is binder's responsibility to ensure that messages stopped being delivered if application is in a shutdown state. There could still be some implication in the stream core, but that is the route I would pick - contact Solace team and the we can review if something needs to be done in core.