search

spring-projects / spring-framework

Spring Framework
https://spring.io/projects/spring-framework
Apache License 2.0
56.64k stars 38.14k forks source link

Shut down task executor beans first during factory destruction [SPR-9512] #14146

Closed spring-projects-issues closed 1 year ago

spring-projects-issues commented 12 years ago

Archie Cobbs opened SPR-9512 and commented

I have a bean factory with this:

    <task:annotation-driven executor="taskExecutor" scheduler="taskScheduler"/>

    <task:executor id="taskExecutor" pool-size="2-10" queue-capacity="100" keep-alive="10"/>
    <task:scheduler id="taskScheduler" pool-size="1"/>

and I'm occasionally seeing this exception on bean factory shutdown:

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'transactionManager':
    Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean
    from a BeanFactory in a destroy method implementation!)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:212)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.determineTransactionManager(TransactionAspectSupport.java:248)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:272)
        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$before$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:63)
        at com.foobar.DefaultQueueProcessor.runQueue(DefaultQueueProcessor.java:50)
        at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)

A scheduled task is starting and then dying because the bean factory is shutting down.

It seems like an improvement would be: before shutting down a bean factory, find all beans that perform scheduled tasks and stop them, so they don't generate new activity during bean factory destruction.

Affects: 3.1.1

9 votes, 12 watchers

spring-projects-issues commented 11 years ago

Thorsten Vogel commented

The same happens in a web environment (with web flow) when shutting down tomcat, running a web application context that has scheduled tasks (same configuration as OP).

Spring 3.2.4

WARN : org.springframework.web.context.support.GenericWebApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'org.springframework.scheduling.config.ContextLifecycleScheduledTaskRegistrar#0': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:210) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:198) at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:155) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:86) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:336) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1049) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1010) at org.springframework.webflow.engine.Flow.destroy(Flow.java:597) at org.springframework.webflow.engine.builder.DefaultFlowHolder.destroy(DefaultFlowHolder.java:100) at org.springframework.webflow.definition.registry.FlowDefinitionRegistryImpl.destroy(FlowDefinitionRegistryImpl.java:114) at org.springframework.webflow.config.FlowRegistryFactoryBean.destroy(FlowRegistryFactoryBean.java:161) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:238) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:510) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:486) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:742) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:455) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1090) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1064) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1010) at org.springframework.web.servlet.FrameworkServlet.destroy(FrameworkServlet.java:794) at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1482) at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1843) at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5614) at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575) at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)

spring-projects-issues commented 9 years ago

Arne Hildebrand commented

We're still facing the issue in the Spring 4.1.4. Through we do have a small fixed delay (1000ms) to trigger our task ...

spring-projects-issues commented 8 years ago

Botez Bianca-Elena commented

This issue is still reproducing in Spring 4.2.3

spring-projects-issues commented 8 years ago

Juergen Hoeller commented

Could you post your specific stacktrace please? Or is it identical to one of the above?

I suppose the "waitForTasksToCompleteOnShutdown" and "awaitTerminationSeconds" settings would help a bit here, but indeed, some beans might get destroyed before the executor/scheduler beans depending the order of bean registration... and if exactly such beans get dynamically obtained from scheduler tasks, the above warning would get logged.

If the registration/shutdown order of beans is indeed the root of the issue, we can try to do something about it within the 4.3 timeframe still.

spring-projects-issues commented 8 years ago

Ike Chen commented

I am facing the same issue.

sequence:

  1. Tomcat server shutdown;
  2. Task in ThreadPoolTaskExecutor continue to run with "waitForTasksToCompleteOnShutdown" and "awaitTerminationSeconds" set;
  3. BeanCreationNotAllowedException thrown because task instance need to be injected with a scoped bean (in my case: scope="step", from spring batch);

I think "shutdown order" might help, with which, I may set my ThreadPoolTaskExecutor to be on the top of the shutdown sequence. Looking forward for this improvement in 4.3 timeframe. Thx.

spring-projects-issues commented 8 years ago

Fabien MARSAUD commented

Same thing with SpringBoot 1.3.3. Though we noticed the issue is systematic on our test environment, and isn't present on our production environment.

TEST: SUSE Linux Enterprise Server 11 (x86_64) patch 1 PROD: Ubuntu 14.04.1 LTS

So, whenever we try to stop a SpringBoot on our TEST environment using SIGTERM (kill -15), the process remains active. Then if we try to call the application (it exposes rest services), we get that output in the browser:

HTTP Status 500 - Request processing failed; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'departureResource': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

type Exception report

message Request processing failed; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'departureResource': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'departureResource': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:980)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:859)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:237)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:112)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
root cause

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'departureResource': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
    org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:216)
    org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    org.springframework.web.method.HandlerMethod.createWithResolvedBean(HandlerMethod.java:245)
    org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:314)
    org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:60)
    org.springframework.web.servlet.handler.AbstractHandlerMapping.getHandler(AbstractHandlerMapping.java:351)
    org.springframework.web.servlet.DispatcherServlet.getHandler(DispatcherServlet.java:1120)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:932)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:859)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:237)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:112)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.32 logs.

Apache Tomcat/8.0.32

On the logs, we have:

2016-04-26 11:55:38.722  INFO 24059 --- [Thread-2] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@2b458293: startup date [Wed Apr 20 12:06:02 GMT 2016]; root of context hierarchy
2016-04-26 11:55:38.725  INFO 24059 --- [Thread-2] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@2635bf84: startup date [Wed Apr 20 12:06:06 GMT 2016]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@2b458293
2016-04-26 11:55:38.728  INFO 24059 --- [Thread-2] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2016-04-26 11:55:38.729  INFO 24059 --- [Thread-2] c.n.e.EurekaDiscoveryClientConfiguration : Unregistering application departure-query with eureka with status DOWN
2016-04-26 11:55:38.742  INFO 24059 --- [Thread-2] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2016-04-26 11:55:38.742  INFO 24059 --- [Thread-2] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans
2016-04-26 11:55:38.743  INFO 24059 --- [Thread-2] o.s.c.n.servo.ServoMetricCollector       : Stopping Servo PollScheduler
2016-04-26 11:55:38.746  INFO 24059 --- [Thread-2] c.n.e.EurekaDiscoveryClientConfiguration : Closing DiscoveryClient.jerseyClient
2016-04-26 11:55:38.747  INFO 24059 --- [Thread-2] c.n.e.EurekaDiscoveryClientConfiguration : Removing application departure-query from eureka
2016-04-26 11:55:38.751  WARN 24059 --- [Thread-2] com.netflix.discovery.DiscoveryClient    : Action: Cancel  => returned status of 404 from http://sagq.deolan.biz:8761/eureka/apps/DEPARTURE-QUERY/sagq
2016-04-26 11:55:38.756 ERROR 24059 --- [Thread-2] com.netflix.discovery.DiscoveryClient    : Can't get a response from http://sagq.deolan.biz:8761/eureka/apps/DEPARTURE-QUERY/sagq
Can't contact any eureka nodes - possibly a security group issue?

java.lang.RuntimeException: Bad status: 404
    at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1155) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryClient.unregister(DiscoveryClient.java:694) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryClient.shutdown(DiscoveryClient.java:684) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryManager.shutdownComponent(DiscoveryManager.java:94) [eureka-client-1.1.147.jar!/:1.1.147]
    at org.springframework.cloud.netflix.eureka.EurekaDiscoveryClientConfiguration.close(EurekaDiscoveryClientConfiguration.java:93) [spring-cloud-netflix-core-1.0.7.RELEASE.jar!/:1.0.7.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_73]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_73]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_73]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_73]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:354) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeDestroyMethods(InitDestroyAnnotationBeanPostProcessor.java:319) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeDestruction(InitDestroyAnnotationBeanPostProcessor.java:153) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:243) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1006) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:982) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:934) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.boot.builder.ParentContextCloserApplicationListener$ContextCloserListener.onApplicationEvent(ParentContextCloserApplicationListener.java:102) [spring-boot-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.boot.builder.ParentContextCloserApplicationListener$ContextCloserListener.onApplicationEvent(ParentContextCloserApplicationListener.java:86) [spring-boot-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:163) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:136) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:381) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:335) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:967) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:901) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]

2016-04-26 11:55:38.757 ERROR 24059 --- [Thread-2] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_DEPARTURE-QUERY/sagq - de-registration failedBad status: 404

java.lang.RuntimeException: Bad status: 404
    at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1155) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryClient.unregister(DiscoveryClient.java:694) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryClient.shutdown(DiscoveryClient.java:684) [eureka-client-1.1.147.jar!/:1.1.147]
    at com.netflix.discovery.DiscoveryManager.shutdownComponent(DiscoveryManager.java:94) [eureka-client-1.1.147.jar!/:1.1.147]
    at org.springframework.cloud.netflix.eureka.EurekaDiscoveryClientConfiguration.close(EurekaDiscoveryClientConfiguration.java:93) [spring-cloud-netflix-core-1.0.7.RELEASE.jar!/:1.0.7.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_73]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_73]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_73]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_73]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:354) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeDestroyMethods(InitDestroyAnnotationBeanPostProcessor.java:319) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeDestruction(InitDestroyAnnotationBeanPostProcessor.java:153) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:243) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979) [spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1006) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:982) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:934) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.boot.builder.ParentContextCloserApplicationListener$ContextCloserListener.onApplicationEvent(ParentContextCloserApplicationListener.java:102) [spring-boot-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.boot.builder.ParentContextCloserApplicationListener$ContextCloserListener.onApplicationEvent(ParentContextCloserApplicationListener.java:86) [spring-boot-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:163) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:136) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:381) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:335) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:967) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:901) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]

2016-04-26 11:55:46.280 ERROR 24059 --- [http-nio-8096-exec-9] s.c.c.d.h.DiscoveryClientHealthIndicator : Error

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'scopedTarget.eurekaDiscoveryClient': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:216) ~[spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35) ~[spring-aop-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.getTarget(CglibAopProxy.java:687) ~[spring-aop-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:637) ~[spring-aop-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at com.netflix.discovery.DiscoveryClient$$EnhancerBySpringCGLIB$$caefaf57.getApplications(<generated>) ~[eureka-client-1.1.147.jar!/:1.1.147]
    at org.springframework.cloud.netflix.eureka.EurekaDiscoveryClient.getServices(EurekaDiscoveryClient.java:133) ~[spring-cloud-netflix-core-1.0.7.RELEASE.jar!/:1.0.7.RELEASE]
    at org.springframework.cloud.client.discovery.health.DiscoveryClientHealthIndicator.health(DiscoveryClientHealthIndicator.java:61) ~[spring-cloud-commons-1.0.5.RELEASE.jar!/:1.0.5.RELEASE]
    at org.springframework.cloud.client.discovery.health.DiscoveryCompositeHealthIndicator$Holder.health(DiscoveryCompositeHealthIndicator.java:52) [spring-cloud-commons-1.0.5.RELEASE.jar!/:1.0.5.RELEASE]
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:81) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:162) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:136) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_73]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_73]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:859) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844) [spring-webmvc-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:237) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:112) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.3.RELEASE.jar!/:1.3.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_73]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_73]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.32.jar!/:8.0.32]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]

We're using Eureka in the app, so it complains too, but basically, this is the same issue as above.

spring-projects-issues commented 8 years ago

Juergen Hoeller commented

As of 4.3 RC2, ScheduledAnnotationBeanPostProcessor features a revised registration mechanism capable of tracking individual bean instances, both on creation for beans of any scope and on destruction of each individual bean. This affects the implicit shutdown order since it is is bound to each bean now, not to ScheduledAnnotationBeanPostProcessor's own lifecycle.

So please revisit this against 4.3 RC2 and see whether there are any further steps to take.

spring-projects-issues commented 6 years ago

Eduard Wirch commented

Spring 5.0.9.RELEASE:

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'myController': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:208)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.web.method.HandlerMethod.createWithResolvedBean(HandlerMethod.java:313)
        at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:325)
        at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:62)
        at org.springframework.web.servlet.handler.AbstractHandlerMapping.getHandler(AbstractHandlerMapping.java:350)
        at org.springframework.web.servlet.DispatcherServlet.getHandler(DispatcherServlet.java:1188)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:964)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:531)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
        at java.base/java.lang.Thread.run(Thread.java:844)
OleksiiBondar commented 5 years ago

Is there any update on this?

limhs25 commented 4 years ago

Not Yet? or Newer? Any body here?

Sineaggi commented 4 years ago

We're seeing this too. Any update?

rustyx commented 4 years ago

Same here with Spring Boot 2.2.4.RELEASE (Spring 5.2.3.RELEASE).

From the looks of it, if a new web request comes in at the exact moment the server is shutting down, the dispatcher tries to re-create the already destroyed controller.

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'mainController': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:208)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.web.method.HandlerMethod.createWithResolvedBean(HandlerMethod.java:336)
        at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.getHandlerInternal(AbstractHandlerMethodMapping.java:368)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.getHandlerInternal(RequestMappingHandlerMapping.java:449)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.getHandlerInternal(RequestMappingHandlerMapping.java:67)
        at org.springframework.web.servlet.handler.AbstractHandlerMapping.getHandler(AbstractHandlerMapping.java:395)
        at org.springframework.web.servlet.DispatcherServlet.getHandler(DispatcherServlet.java:1234)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1016)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
acsipak commented 3 years ago

Any suggested workarounds?

stefanocke commented 3 years ago

As far as I see, there are at least two issues here in the comments:

  1. BeanCreationNotAllowedException in scheduled tasks
  2. BeanCreationNotAllowedException in web requests

Regarding scheduled tasks:

I also experienced BeanCreationNotAllowedExceptions but sometimes also "EntityManager is closed", if the task did some JPA work. When I looked deeper into this, I recognized the main cause was, that my task was not interruptible.

So, the following happened:

  1. TaskScheduler is stopped and tries to interrupt its scheduled tasks
  2. Tasks continue to run, since they are not interruptible.
  3. Spring shuts down other beans, like EntityManagerFactory
  4. Task try to get beans / resources, that no longer exist -> Exceptions are thrown.

By "not interruptible" I mean, they are not running any oprations mentionend in java.lang.Thread.interrupt(). This is for instance even true if the task does lots of DB work and uses Postgres JDBC driver, since it is pre-NIO. (This explanation is probably not perfect, but I am quite sure my long running DB tasks were definitely not interruptible...)

Thus, a possible solution for scheduled tasks is to make them interruptible b checking for Thread.current().isInterrupted() frequently. See for instance https://www.ibm.com/developerworks/java/library/j-jtp05236/index.html?ca=drs-#2.2 (Section: "Implementing cancelable tasks")

Additionally, you should set spring.task.execution.shutdown.await-termination-period and spring.task.execution.shutdown.await-termination-period. So you compensate for the milliseconds or seconds you scheduled task needs until it checks for Thread.current().isInterrupted() the next time.

Update 2021-02-03 While this appraoach helps me in most cases, it is unfortunately not complete: If my tasks publish Application-Events, I get a similar Exception as mentioned by Thorsten Vogel. Please see my next comment for further analysis.

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'refreshEventListener': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:207)
    at org.springframework.context.event.AbstractApplicationEventMulticaster.retrieveApplicationListeners(AbstractApplicationEventMulticaster.java:247)
    at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:204)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:134)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:404)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:374)

Regarding web requests:

I have not too much experience with this case but hope that https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.3-Release-Notes#graceful-shutdown would help.

stefanocke commented 3 years ago

Some more debugging... I looked at the disposable bean names and their order by debugging into destroySingletons() . For better overview I only show some relevant beans here:

[... dataSource,
 ..., entityManagerFactory, 
 ..., taskScheduler,
 ... , applicationTaskExecutor, 
 ..., refreshEventListener, 
..., springApplicationAdminRegistrar, ... ]

The destruction oder is from last to first element. As you can see, dataSource and entityManagerFactory are already in good positions. If my tasks depend on them, I can rely on them being destroyed after taskScheduler and applicationTaskExecutor.

However, I got my exception for the bean "refreshEventListener" and also for "springApplicationAdminRegistrar" (which I do not use directly). As the stacktrace in my previous comment shows, they are just requested because the EventMulticaster asks for all potential EventListener beans. (AFAIU. Excuse me, if incorrect...) So, my tasks have a very "indirect" dependency on those beans, just because I publish events. I tend to belive it is not possible to completely detect all those kinds of dependencies.

How could Spring or Spring Boot help with this?

Workaround

Besides the already mentioned

my workaround for this part of the issue is to shutdown the TaskExecuter and TaskScheduler early by myself. I guess, this is not completely surprising and there are similar solutions out there on SO:

@Bean
    public ApplicationListener<ContextClosedEvent> earlyTaskExecutorShutdown(ThreadPoolTaskExecutor applicationTaskExecutor) {
        return new ApplicationListener<ContextClosedEvent>() {

            @Override
            public void onApplicationEvent(ContextClosedEvent event) {
                applicationTaskExecutor.shutdown();
            }

        };
    }

@Bean
    public ApplicationListener<ContextClosedEvent> earlyTaskSchedulerShutdown(ThreadPoolTaskScheduler taskScheduler) {
        return new ApplicationListener<ContextClosedEvent>() {

            @Override
            public void onApplicationEvent(ContextClosedEvent event) {
                taskScheduler.shutdown();
            }

        };
    }
jhoeller commented 1 year ago

Superseded by #27090 now, being released today in 6.1 M2.

We're effectively doing what's shown above, triggering an early shutdown on ContextClosedEvent for both ThreadPoolTaskExecutor and ThreadPoolTaskScheduler, as well as for ScheduledAnnotationBeanPostProcessor.