spring-cloud / spring-cloud-netflix

Integration with Netflix OSS components
http://cloud.spring.io/spring-cloud-netflix/
Apache License 2.0
4.87k stars 2.44k forks source link

Use spring-clould-zuul send request timeout #3172

Closed ljee-hash closed 6 years ago

ljee-hash commented 6 years ago

I use zuul send request the response is waitting ,

I have rewrited the SendResponseFilter ,because in my project I want do cache and convert result to JSON and JsonP .

by analysis the dump , I got the error !

"http-nio-8080-exec-1" #86 daemon prio=5 os_prio=0 tid=0x0000000059f95800 nid=0x4970 waiting on condition [0x000000005fc1c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f68e5218> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:139)
        at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:307)
        at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:65)
        at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:193)
        at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:186)
        at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:108)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forwardRequest(SimpleHostRoutingFilter.java:342)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forward(SimpleHostRoutingFilter.java:305)
        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:179)
        at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
        at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197)
        at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:161)
        at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:120)
        at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
        at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
        at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
        at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
        at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequestInternal(ZuulController.java:43)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:522)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1110)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        - locked <0x00000000f22e64b0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x00000000f326c5a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

## rewrite method **writeResponse()**

```java
    private void writeResponse() throws Exception {
        RequestContext context = RequestContext.getCurrentContext();
        // there is no body to send
        if (context.getResponseBody() == null
                && context.getResponseDataStream() == null) {
            return;
        }
        HttpServletResponse servletResponse = context.getResponse();
        if (servletResponse.getCharacterEncoding() == null) { // only set if not set
            servletResponse.setCharacterEncoding("UTF-8");
        }
        OutputStream outStream = servletResponse.getOutputStream();
        InputStream is = null;
        try {
            if (RequestContext.getCurrentContext().getResponseBody() != null) {
                String body = RequestContext.getCurrentContext().getResponseBody();
                writeResponse(
                        new ByteArrayInputStream(
                                body.getBytes(servletResponse.getCharacterEncoding())),
                        outStream);
                return;
            }
            boolean isGzipRequested = false;
            final String requestEncoding = context.getRequest()
                    .getHeader(ZuulHeaders.ACCEPT_ENCODING);

            if (requestEncoding != null
                    && HTTPRequestUtils.getInstance().isGzipped(requestEncoding)) {
                isGzipRequested = true;
            }
            is = context.getResponseDataStream();
            //handle the jsonp

//            is = this.rewriteResponse(context); //the origin code 

            if(context.getRequest() != null){
                Map<String,Object> resultMap = new LinkedHashMap<>();
                HttpServletRequest request = context.getRequest();
                String callback = request.getParameter("callback");
                if (context.getResponseStatusCode() == HttpStatus.SC_OK) {
//                    String responseBody = CharStreams.toString(new InputStreamReader(context.getResponseDataStream()));
                    String responseBody = IOUtils.toString(context.getResponseDataStream());
                    resultMap = JSON.parseObject(responseBody, LinkedHashMap.class);

                }
                if(resultMap != null && resultMap.size() > 0){
                    resultMap.put("ret",context.getResponseStatusCode());
                    if (StringUtils.isNotEmpty(callback)) {
                        StringBuffer buf = new StringBuffer();
                        buf.append(callback);
                        buf.append("(");
                        buf.append(JsonUtil.toJSON(resultMap));
                        buf.append(")");
                        is = new ByteArrayInputStream(buf.toString().getBytes(StandardCharsets.UTF_8));
                    }
                }
            }

            InputStream inputStream = is;
            if (is != null) {
                if (context.sendZuulResponse()) {
                    // if origin response is gzipped, and client has not requested gzip,
                    // decompress stream
                    // before sending to client
                    // else, stream gzip directly to client
                    if (context.getResponseGZipped() && !isGzipRequested) {
                        // If origin tell it's GZipped but the content is ZERO bytes,
                        // don't try to uncompress
                        final Long len = context.getOriginContentLength();
                        if (len == null || len > 0) {
                            try {
                                inputStream = new GZIPInputStream(is);
                            }
                            catch (java.util.zip.ZipException ex) {
                                log.debug(
                                        "gzip expected but not "
                                                + "received assuming unencoded response "
                                                + RequestContext.getCurrentContext()
                                                .getRequest().getRequestURL()
                                                .toString());
                                inputStream = is;
                            }
                        }
                        else {
                            // Already done : inputStream = is;
                        }
                    }
                    else if (context.getResponseGZipped() && isGzipRequested) {
                        servletResponse.setHeader(ZuulHeaders.CONTENT_ENCODING, "gzip");
                    }
                    writeResponse(inputStream, outStream);
                }
            }
        }
        finally {
            try {
                if (is != null) {
                    is.close();
                }
                outStream.flush();
                // The container will close the stream for us
            }
            catch (IOException ex) {
            }
        }
    }

the application.properties

ribbon.eureka.enabled=false
zuul.host.socket-timeout-millis=6000
zuul.host.connect-timeout-millis=3000
server.port=8080
zuul.SendResponseFilter.post.disable=true

# Disable Hystrix timeout globally (for all services)
hystrix.command.default.execution.timeout.enabled=false
# Disable Hystrix timeout for a single service
#hystrix.command.<serviceName>.execution.timeout.enabled=false
# Increase the Hystrix timeout to 60s (globally)
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=6000
# Increase the Hystrix timeout to 60s (per service)
ryanjbaxter commented 6 years ago

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

ljee-hash commented 6 years ago

thank you very much! I have solve the issues,it's my fault.

 String responseBody = CharStreams.toString(new InputStreamReader(context.getResponseDataStream()));
                    String responseBody = IOUtils.toString(context.getResponseDataStream());// replaced by IOUtils.copy(writer,inputStream)

the two ways is not thread safe ! I forget it!

I solved it by rewriting the synchronous Writer method.But I don't know why asynchronous methods cause problems.

            if(context.getRequest() != null){
                Map<String,Object> resultMap = new LinkedHashMap<>();
                HttpServletRequest request = context.getRequest();
                String callback = request.getParameter("callback");
                if (context.getResponseStatusCode() == HttpStatus.SC_OK) {

//                    String responseBody = CharStreams.toString(new InputStreamReader(context.getResponseDataStream()));
                    String responseBody = IOUtils.toString(context.getResponseDataStream());
                    resultMap = JSON.parseObject(responseBody, LinkedHashMap.class);

                }
                if(resultMap != null && resultMap.size() > 0){
                    resultMap.put("ret",context.getResponseStatusCode());
                    if (StringUtils.isNotEmpty(callback)) {
                        StringBuffer buf = new StringBuffer();
                        buf.append(callback);
                        buf.append("(");
                        buf.append(JsonUtil.toJSON(resultMap));
                        buf.append(")");
                        is = new ByteArrayInputStream(buf.toString().getBytes(StandardCharsets.UTF_8));
                    }
                }
            }