search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

instanceInfoReplicator start after status change, then RejectedExecutionException #2267

Closed wangwenyao closed 7 years ago

wangwenyao commented 7 years ago

https://github.com/Netflix/eureka/issues/983

ryanjbaxter commented 7 years ago

Please copy the information from the Netflix issue into this issue instead of just providing a link. Also a sample application to reproduce the issue would be welcome. Finally please learn how to format code in GitHub issues https://help.github.com/articles/creating-and-highlighting-code-blocks/

wangwenyao commented 7 years ago

spring cloud bus with kafka, when send "bus/refresh", occur RejectedExecutionException

pom:

  <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-eureka</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-bus-kafka</artifactId>
        </dependency>

log:

[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.365 o.s.c.a.AnnotationConfigApplicationContext[583] - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@22efb7d7: startup date [Tue Sep 19 13:16:46 CST 2017]; root of context hierarchy
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.431 o.s.b.f.a.AutowiredAnnotationBeanPostProcessor[155] - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.436 o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker[325] - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$e1e26bfa] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.728 o.s.b.SpringApplication[593] - No active profile set, falling back to default profiles: default
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.736 o.s.c.a.AnnotationConfigApplicationContext[583] - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@714ab7c1: startup date [Tue Sep 19 13:16:46 CST 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@22efb7d7
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.741 o.s.b.f.a.AutowiredAnnotationBeanPostProcessor[155] - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.755 o.s.b.SpringApplication[57] - Started application in 0.686 seconds (JVM running for 490.153)
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:46.756 o.s.c.a.AnnotationConfigApplicationContext[984] - Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@714ab7c1: startup date [Tue Sep 19 13:16:46 CST 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@22efb7d7
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.236 c.n.d.DiscoveryClient[852] - Shutting down DiscoveryClient ...
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.238 o.s.c.n.e.InstanceInfoFactory[70] - Setting initial instance status as: STARTING
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.244 c.n.d.DiscoveryClient[884] - Unregistering ...
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.252 c.n.d.DiscoveryClient[886] - DiscoveryClient_CONFIG-CENTER/192.168.50.150:8102 - deregister  status: 200
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.264 c.n.d.DiscoveryClient[873] - Completed shut down of DiscoveryClient
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.273 c.n.d.DiscoveryClient[344] - Initializing Eureka in region us-east-1
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.307 c.n.d.p.DiscoveryJerseyProvider[70] - Using JSON encoding codec LegacyJacksonJson
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.308 c.n.d.p.DiscoveryJerseyProvider[71] - Using JSON decoding codec LegacyJacksonJson
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.310 c.n.d.p.DiscoveryJerseyProvider[80] - Using XML encoding codec XStreamXml
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.310 c.n.d.p.DiscoveryJerseyProvider[81] - Using XML decoding codec XStreamXml
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.598 c.n.d.s.r.a.ConfigClusterResolver[43] - Resolving eureka endpoints via configuration
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.602 c.n.d.DiscoveryClient[1244] - Starting heartbeat executor: renew interval is: 10
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.604 c.n.d.InstanceInfoReplicator[59] - InstanceInfoReplicator onDemand update allowed rate per min is 4
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.609 c.n.d.DiscoveryClient[425] - Discovery Client initialized at timestamp 1505798207609 with initial instances count: 0
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.615 o.s.c.n.e.s.EurekaServiceRegistry[64] - Unregistering application config-center with eureka with status DOWN
[http-nio-8102-exec-6] INFO  2017-09-19 13:16:47.662 o.s.c.n.e.s.EurekaServiceRegistry[40] - Registering application config-center with eureka with status UP
[http-nio-8102-exec-6] WARN  2017-09-19 13:16:47.664 c.n.d.DiscoveryClient[1277] - Saw local status change event StatusChangeEvent [timestamp=1505798207664, current=UP, previous=DOWN]
[DiscoveryClient-InstanceInfoReplicator-0] INFO  2017-09-19 13:16:47.666 c.n.d.DiscoveryClient[795] - DiscoveryClient_CONFIG-CENTER/192.168.50.150:8102: registering service...
[http-nio-8102-exec-6] ERROR 2017-09-19 13:16:47.680 o.a.c.c.C.[.[.[.[dispatcherServlet][181] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3acfc46b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@4213732f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5]] with root cause
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3acfc46b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@4213732f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
    at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
    at com.netflix.discovery.InstanceInfoReplicator.onDemandUpdate(InstanceInfoReplicator.java:77)
    at com.netflix.discovery.DiscoveryClient.registerHealthCheck(DiscoveryClient.java:618)
    at com.netflix.discovery.DiscoveryClient$$FastClassBySpringCGLIB$$a84c8cb4.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.cloud.context.config.StandardBeanLifecycleDecorator$2.invoke(StandardBeanLifecycleDecorator.java:85)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at org.springframework.cloud.netflix.eureka.CloudEurekaClient$$EnhancerBySpringCGLIB$$c54e5.registerHealthCheck(<generated>)
    at org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry.register(EurekaServiceRegistry.java:49)
    at org.springframework.cloud.netflix.eureka.serviceregistry.EurekaAutoServiceRegistration.start(EurekaAutoServiceRegistration.java:73)
    at org.springframework.cloud.netflix.eureka.EurekaDiscoveryClientConfiguration$EurekaClientConfigurationRefresher.onApplicationEvent(EurekaDiscoveryClientConfiguration.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:256)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:177)
    at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:140)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
    at org.springframework.cloud.context.scope.refresh.RefreshScope.refreshAll(RefreshScope.java:146)
    at org.springframework.cloud.context.refresh.ContextRefresher.refresh(ContextRefresher.java:58)
    at org.springframework.cloud.bus.event.RefreshListener.onApplicationEvent(RefreshListener.java:43)
    at org.springframework.cloud.bus.event.RefreshListener.onApplicationEvent(RefreshListener.java:30)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
    at org.springframework.cloud.bus.endpoint.AbstractBusEndpoint.publish(AbstractBusEndpoint.java:48)
    at org.springframework.cloud.bus.endpoint.RefreshBusEndpoint.refresh(RefreshBusEndpoint.java:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
ryanjbaxter commented 7 years ago

Does this happen every time? It seems like it might be a timing issue and the thread pool is being terminated due to the refresh event.

ryanjbaxter commented 7 years ago

This appears to be the same as #2228

hao0111 commented 6 years ago

Is it fixed? Using spring cloud E.version,it is also happen!!

ryanjbaxter commented 6 years ago

No need to spam, you already saw #2228 is still open