search

spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

spring cloud zuul : could not acquire a semaphore for execution and no fallback available. #2064

Closed robertYang1024 closed 7 years ago

robertYang1024 commented 7 years ago

I used Spring cloud zuul as a gateway, but always reported a error , the loadrunner concurrency is 200 , and i set ` zuul.eureka.MSG-SERVICE.semaphore.maxSemaphores: 2000 , but still reported errors . com.netflix.zuul.exception.ZuulException: Forwarding error at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:158) ~[spring-cloud-netflix-core-1.1.4.RELEASE.jar!/:1.1.4.RELEASE] at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:133) ~[spring-cloud-netflix-core-1.1.4.RELEASE.jar!/:1.1.4.RELEASE] at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:79) ~[spring-cloud-netflix-core-1.1.4.RELEASE.jar!/:1.1.4.RELEASE] at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112) ~[zuul-core-1.1.0.jar!/:1.1.0] at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197) ~[zuul-core-1.1.0.jar!/:1.1.0] at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:161) ~[zuul-core-1.1.0.jar!/:1.1.0] at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:120) ~[zuul-core-1.1.0.jar!/:1.1.0] at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.1.0.jar!/:1.1.0] at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.1.0.jar!/:1.1.0] at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.1.0.jar!/:1.1.0] at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:158) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequestInternal(ZuulController.java:43) [spring-cloud-netflix-core-1.1.4.RELEASE.jar!/:1.1.4.RELEASE] at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar!/:3.1.0] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843) [spring-webmvc-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar!/:3.1.0] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:281) [spring-boot-actuator-1.3.6.RELEASE.jar!/:1.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:156) [spring-cloud-sleuth-core-1.0.3.RELEASE.jar!/:1.0.3.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:115) [spring-boot-actuator-1.3.6.RELEASE.jar!/:1.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.6.RELEASE.jar!/:1.3.6.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175) [undertow-servlet-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:802) [undertow-core-1.3.23.Final.jar!/:1.3.23.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95] Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: MSG-SERVICE could not acquire a semaphore for execution and no fallback available. at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:783) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:768) ~[hystrix-core-1.5.3.jar!/:1.5.3] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.5.jar!/:1.1.5] at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1448) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1373) ~[hystrix-core-1.5.3.jar!/:1.5.3] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.observers.Subscribers$5.onError(Subscribers.java:224) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.Observable.unsafeSubscribe(Observable.java:8460) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.Observable.unsafeSubscribe(Observable.java:8460) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:50) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.Observable.subscribe(Observable.java:8553) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.Observable.subscribe(Observable.java:8520) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:57) ~[rxjava-1.1.5.jar!/:1.1.5] at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:401) ~[rxjava-1.1.5.jar!/:1.1.5] at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:373) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:329) ~[hystrix-core-1.5.3.jar!/:1.5.3] at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:127) ~[spring-cloud-netflix-core-1.1.4.RELEASE.jar!/:1.1.4.RELEASE] ... 74 common frames omitted Caused by: java.lang.RuntimeException: could not acquire a semaphore for execution at com.netflix.hystrix.AbstractCommand.handleSemaphoreRejectionViaFallback(AbstractCommand.java:927) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:529) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.AbstractCommand.access$100(AbstractCommand.java:59) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:426) ~[hystrix-core-1.5.3.jar!/:1.5.3] at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:423) ~[hystrix-core-1.5.3.jar!/:1.5.3] at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46) ~[rxjava-1.1.5.jar!/:1.1.5] ... 92 common frames omitted`

Thank you in Advance

ryanjbaxter commented 7 years ago

I think we might need some more information, can you provide a sample that reproduces this?

wenyuguang commented 7 years ago

hi.I have a similar situation, I was testing a api-gateway concurrency, when the number of concurrent occurs more than 100 of the error, I opened the debug log and found that the Tomcat error is null pointer.

`2017-06-22 15:19:41.420 DEBUG 16132 --- [io-5555-exec-47] o.a.http.impl.execchain.MainClientExec : Connection can be kept alive indefinitely 2017-06-22 15:19:41.420 DEBUG 16132 --- [on(9)-127.0.0.1] sun.rmi.transport.tcp : RMI TCP Connection(9)-127.0.0.1: (port 53344) op = 84 2017-06-22 15:19:41.420 DEBUG 16132 --- [15)-150.0.32.36] sun.rmi.transport.tcp : RMI TCP Connection(15)-150.0.32.36: (port 53346) op = 82 2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-47] h.i.c.PoolingHttpClientConnectionManager : Connection [id: 46][route: {}->http://150.0.2.167:2222] can be kept alive indefinitely 2017-06-22 15:19:41.403 DEBUG 16132 --- [io-5555-exec-73] o.s.c.n.zuul.filters.SimpleRouteLocator : route matched=ZuulProperties.ZuulRoute(id=api-test, path=/api-test/**, serviceId=service-test, url=null, stripPrefix=true, retryable=null, sensitiveHeaders=[], customSensitiveHeaders=false) 2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-27] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header

java.io.EOFException: null at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1242) ~[tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1182) ~[tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:713) ~[tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:362) ~[tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:683) ~[tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861) [tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455) [tomcat-embed-core-8.5.15.jar:8.5.15] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.15.jar:8.5.15] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_111] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.15.jar:8.5.15] at java.lang.Thread.run(Unknown Source) [na:1.8.0_111]

2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-27] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@707a674f:org.apache.tomcat.util.net.NioChannel@4469fb6c:java.nio.channels.SocketChannel[connected local=/127.0.0.1:5555 remote=/127.0.0.1:53780]], Status in: [OPEN_READ], State out: [CLOSED] 2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-72] h.i.c.PoolingHttpClientConnectionManager : Connection request: [route: {}->http://150.0.2.167:2222][total kept alive: 69; route allocated: 44 of 50; total allocated: 86 of 200] 2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-73] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'ribbonLoadBalancingHttpClient' 2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-73] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'ribbonLoadBalancer' 2017-06-22 15:19:41.421 DEBUG 16132 --- [io-5555-exec-73] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'ribbonClientConfig'`

ryanjbaxter commented 7 years ago

@wenyuguang I dont see how your problem is related.

robertYang1024 commented 7 years ago

@ryanjbaxter i wrote a message service which named MSG-SERVICE , and i called the api of the message service through the gateway ，When I used loadrunner to test 200 concurrent times , the zuul will show that error
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: MSG-SERVICE could not acquire a semaphore for execution and no fallback available.

i set
zuul.eureka.MSG-SERVICE.semaphore.maxSemaphores: 2000； but looks like it does not work . any other information do you want to know ?

spencergibb commented 7 years ago

Two things, how long do you requests take? Also, you could try using a thread pool.

harishkadamudi commented 7 years ago

Even I am running into a similar Issue. Testing concurrency using zuul as api gateway with 150 requests/sec(jmeter).

I have a client called Order-service, which takes close to 4 secs to serve the response.

Thread.sleep(4000)

Here is my configuration

hystrix:
  command:
    default:
      execution:
        isolation:
          strategy: THREAD
          thread:
            timeoutInMilliseconds: 40000
ribbon:
  MaxHttpConnectionsPerHost: 200
  okToRetryOnAllErrors: true 
  OkToRetryOnAllOperations: true
  MaxAutoRetries: 1
  MaxAutoRetriesNextServer: 1
  ReadTimeout: 6000
  ConnectTimeout: 6000

Exception 

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: order-service could not acquire a semaphore for execution and no fallback available.
    at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:822) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:807) ~[hystrix-core-1.5.10.jar:1.5.10]
    at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar:1.1.10]
    at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1472) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1397) ~[hystrix-core-1.5.10.jar:1.5.10]
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.observers.Subscribers$5.onError(Subscribers.java:230) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.unsafeSubscribe(Observable.java:10211) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.subscribe(Observable.java:10307) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.Observable.subscribe(Observable.java:10274) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:412) ~[rxjava-1.1.10.jar:1.1.10]
    at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344) ~[hystrix-core-1.5.10.jar:1.5.10]
    at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:152) ~[spring-cloud-netflix-core-1.3.0.RELEASE.jar:1.3.0.RELEASE]
    ... 67 common frames omitted
Caused by: java.lang.RuntimeException: could not acquire a semaphore for execution
    at com.netflix.hystrix.AbstractCommand.handleSemaphoreRejectionViaFallback(AbstractCommand.java:966) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:554) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413) ~[hystrix-core-1.5.10.jar:1.5.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46) ~[rxjava-1.1.10.jar:1.1.10]
    ... 91 common frames omitted
harishkadamudi commented 7 years ago

If I decrease thread size to 100. I see no Issues.

ryanjbaxter commented 7 years ago

@harishkadamudi can you be more specific about what u changed?

harishkadamudi commented 7 years ago

As I described, was using jMeter to test the concurrency. I decreased no. of threads in jMeter to 100.

harishkadamudi commented 7 years ago

@ryanjbaxter , Issue is not intermittent, I could simulate it consistently

spencergibb commented 7 years ago

to change to using threads in zuul you need to set zuul.ribbonIsolationStrategy=THREAD.

harishkadamudi commented 7 years ago

Setting above property didn't help me, In fact Issue got worsen further. Now requests are failing for 10+ request itself.

Below is the exception trace.

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26c4f530 rejected from java.util.concurrent.ThreadPoolExecutor@353d37f0[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 0]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[na:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [na:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [na:1.8.0_102]
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[na:1.8.0_102]
    at com.netflix.hystrix.strategy.concurrency.HystrixContextScheduler$ThreadPoolWorker.schedule(HystrixContextScheduler.java:172) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.strategy.concurrency.HystrixContextScheduler$HystrixContextSchedulerWorker.schedule(HystrixContextScheduler.java:106) ~[hystrix-core-1.5.10.jar:1.5.10]
    at rx.internal.operators.OperatorSubscribeOn.call(OperatorSubscribeOn.java:45) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OperatorSubscribeOn.call(OperatorSubscribeOn.java:30) ~[rxjava-1.1.10.jar:1.1.10]
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.1.10.jar:1.1.10]
    ... 113 common frames omitted

But certainly I don't see "could not acquire a semaphore for execution"

harishkadamudi commented 7 years ago

this could be the reason. Spring Netflix forces semaphore Isolation.

@spencergibb @ryanjbaxter , By any chance can't we force to use Thread? If you brief us why forcing to semaphore that will help for better understanding.

harishkadamudi commented 7 years ago

wasn't this resolved as part of this, where we can customize Isolation strategy and force to use Thread.

harishkadamudi commented 7 years ago

changed Strategy back to THREAD, now I am facing below exception for 100 concurrent users.

Caused by: java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
    at com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:979) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:557) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419) ~[hystrix-core-1.5.10.jar:1.5.10]
    at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413) ~[hystrix-core-1.5.10.jar:1.5.10]
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46) ~[rxjava-1.1.10.jar:1.1.10]
    ... 91 common frames omitted
harishkadamudi commented 7 years ago

When using isolation strategy THREAD, we need to set thread pool size, set hystrix.threadpool.coreSize to desired number. Things will be fine.

ryanjbaxter commented 7 years ago

@harishkadamudi sorry for not being very responsive, I have been getting some other work done. Sounds like you have the right configuration now. Are there any issues still left unresolved

xz43 commented 7 years ago

Try: zuul.semaphore.max-semaphores=2000