search

Netflix / Hystrix

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.
24.07k stars 4.7k forks source link

Dashboard: unable to connect before first Hystrix request is handled (on Cloud Foundry) #1117

Open C-Otto opened 8 years ago

C-Otto commented 8 years ago

This is very similar to #85.

I am using Hystrix dashboard 1.5.0 (running the .war file in Cloud Foundry on Tomcat 8), and an application with Hystrix Core/Metrics Event Stream 1.4.23 (same cloud). For this test I am not using Turbine.

The dashboard usually works as expected, but not in this case:

At first I see "Loading ..." and, after a few reloads, "Unable to connect to Command Metric Stream.".

The log of the dashboard just shows Proxy opening connection to: https://URL/hystrix.stream.

Sadly, actually using Hystrix inside the application does not fix the problem, as the first five (failed) connections seem to block new connection attempts.

mattrjacobs commented 8 years ago

Can you try using the same versions of both dashboard and the server-side libraries? If that doesn't work, please send me the browser console and I'll take a look

C-Otto commented 8 years ago

@mattrjacobs I can reproduce this as described with 1.5.1 (core, event stream, dashboard).

C-Otto commented 8 years ago

The console is empty for the first few tries ("Loading..."). In the "Unable to connect" case I get:

monitor.html?streams=[{"name"%3A""%2C"stream"%3A"https%3A%2F%2FURL…:1 EventSource's response has a MIME type ("text/plain") that is not "text/event-stream". Aborting the connection.
monitor.html?streams=[{"name"%3A""%2C"stream"%3A"https%3A%2F%2FURL…:119 Connection was closed on error: [object Event]
C-Otto commented 8 years ago

Using my browser I get for up to five (re)loads a text/stream showing ping messages. Afterwards I get

HTTP Status 503 - MaxConcurrentConnections reached: 5
type Status report

message MaxConcurrentConnections reached: 5

description The requested service is not currently available.

Apache Tomcat/8.0.32
C-Otto commented 8 years ago

Workaround (called during initalization):

import com.netflix.hystrix.HystrixCommand;
import com.netflix.hystrix.HystrixCommandGroupKey;

// Workaround for https://github.com/Netflix/Hystrix/issues/1117
class HystrixDummy {
    static void start() {
        new DummyCommand().execute();
    }

    private static class DummyCommand extends HystrixCommand<Void> {
        DummyCommand() {
            super(HystrixCommandGroupKey.Factory.asKey("Dummy"));
        }

        @Override
        protected Void run() throws Exception {
            return null;
        }
    }
}
mattrjacobs commented 8 years ago

I just tested and cannot reproduce using Tomcat7/Tomcat8.

The console log indicates that it's not getting text/event-stream HTTP content-type back. Can you try curling the event-stream URL and send what the headers are?

When I do it, I get: 

< HTTP/1.1 200 OK
< Content-Type: text/event-stream
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Transfer-Encoding: chunked
< Server: xxx

And putting that same event-stream URL in the dashboard starts filling the dashboard up with data.

C-Otto commented 8 years ago

Just to clarify, the content-type error occurs when the error message appears in the browser. I don't think that this is something worth investigating (which I can only do next tuesday), as the error itself is the problem.

The real issue happens before: the ping-replies (I assume (!) these are returned as a correct stream) do not cause the "Loading..." to disappear in the browser, and somehow cause a lock-up in the server.

Are you sure that no single Hystrix command was executed before you started the dashboard?

mattrjacobs commented 8 years ago

Yeah, I missed that bit of context. I was testing with a running system that had already started executing commands. Let me see what can be done about the case in which you connect to a system that has executed 0 commands

mattrjacobs commented 8 years ago

OK, here's where I've gotten to.

This works:

This does not work:

I don't know enough about the front-end/browser space to give you a technical solution for that problem. My solution would be to not start a dashboard/consuming data until an instance is fully started UP. Within Netflix, we use Eureka as a central registry for nodes, and don't inform metrics consumers about instances until they register as UP in Eureka.

Hope that helps!

C-Otto commented 8 years ago

The real issue is that the server side enters a broken state where your "fix" does not help. Instead of executing a command (to work around the problem, as you described) the dashboard might be reloaded a few times. In the latter case any new connection attempt fails, no matter if commands are executed or not.

To summarize, this is not helpful. I already mentioned a workaround, but the bug still is valid. I expect the dashboard to show something meaningful (an empty page?) when I try to connect to a 'fresh' (no command executed) instance instead of blocking or putting the server into a broken state. This is especially important as this is the most simple case I can come up with, and this is exactly what I intended to show in a dashboard demonstration for other programmers.

mattrjacobs commented 8 years ago

I tried to replicate the steps in your first comment with the hystrix-example-webapp.

Did I miss anything that's relevant to your bug-report?

Here are the dashboard Tomcat logs on the 20x reload, demonstrating that there are no connection problems on reload:

2016-03-07 09:54:45 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:49 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:50 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:51 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:52 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:53 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:54 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:55 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:56 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:57 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:58 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:54:59 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:00 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:01 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:02 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:03 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:04 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:05 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:06 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

2016-03-07 09:55:07 INFO  com.netflix.hystrix.dashboard.stream.ProxyStreamServlet:93 [ProxyStreamServlet] [doGet]:

Proxy opening connection to: http://localhost:8989/hystrix-examples-webapp/hystrix.stream
C-Otto commented 8 years ago

The "should" definitions are fine. The last two bullet points are where the expectations are not met in my case. In the sixth (fifth?) reload attempt I don't see "Loading..." anymore, but the error message. Could this be related to a Cloud Foundry routing problem? I'll try again tomorrow without Cloud Foundry in between (just localhost).

C-Otto commented 8 years ago

I cannot reproduce this with both app and dashboard running on localhost. I assume that on Cloud Foundry the (five) connections are kept in an open state, even though the browser at the other end reloaded the page in between. Is this something worth investigating? Should the dashboard work nevertheless (i.e., is this still a valid bug report)?

C-Otto commented 8 years ago

Reproducible with the dashboard running locally, the app running in the cloud.

C-Otto commented 8 years ago

After 15 minutes the connections are freed again, one by one:

IOException while trying to write (generally caused by client disconnecting). Will stop polling.

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

C-Otto commented 8 years ago

Related: https://github.com/cloudfoundry/gorouter/issues/71

agentgt commented 8 years ago

I looked at the HystrixStreamServlet and I'm surprised more people are not having issues as I believe it will eventually break for many servlet containers / reverse proxies particularly if the client fails to let the server know that it disconnected.

This happens because the server keeps a single HTTP 1.1 connection open via a single request (the client is not polling but the server is) and continues to write to it by polling a queue. While in theory this works I have found issues doing this fake comet push style with load balancers / reverse proxies as well as the servlet container failing to clean up connections and timing out.

If another protocol like websockets are used many load balancers are smart enough not to close connections. Some load balancers also look for SSE and respect that as well. Regardless the StreamServlet timeout exceptions are probably not going to go away unless the HystrixStreamServlet is ported to Servlet 3.0 async style.

mattrjacobs commented 8 years ago

Yeah, this is a good idea. Opened #1123

cmittendorf commented 8 years ago

This problem also occured with an application deployed on AWS Elasticbeanstalk. We were able to use the monitor on our stage environment running on a single machine. But when we deployed on live (more than one backend machine) the monitor failed to work with the described error.

caarlos0 commented 8 years ago

I get this error with Jetty 9 too.

mattrjacobs commented 8 years ago

@C-Otto I will try deploying an application that loads the metrics servlet and never executes Hystrix commands, and runs on AWS. That should narrow to a CloudFoundry-specific issue or not.

C-Otto commented 8 years ago

@mattrjacobs any news?

mattrjacobs commented 8 years ago

@C-Otto Not yet, I'm sorry to say. Will try to find time shortly

C-Otto commented 8 years ago

@mattrjacobs ping :)

samaira-singh commented 8 years ago

I am also facing the same issue whenever I try to access hystrix.stream with http://localhost:8080/hystrix.stream or http://localhost:8080/ApplicationName/hystrix.stream I am using jetty 9 server. logs below: Can anyone Please let me know the solution to fix this.

2016-07-01 14:37:16.347 [qtp2071592732-25] INFO [3ac1a5b8-dec7-455f-b988-75580375e307] --- ashboardConfiguration$ProxyStreamServlet :

Proxy opening connection to: http://localhost:8080/hystrix.stream

2016-07-01 14:37:26.461 [qtp2071592732-23] ERROR [29178791-95ad-4548-b9da-9e69ae651fd3] --- ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://localhost:8080/hystrix.stream java.net.SocketTimeoutException: Read timed out

2016-07-02 17:22:53.110 [qtp1332104174-21] INFO [6623ea68-c2c5-4c0a-8d6e-0dec0cbaed5a] --- ashboardConfiguration$ProxyStreamServlet :

Proxy opening connection to: http://localhost:8080/Application-Name/hystrix.stream

gangchen03 commented 8 years ago

Upgrading to the latest hystrix libraries (including the dashboard) seems resolving the issue. https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-hystrix-dashboard/1.1.3.RELEASE However, this works if the Hystrix dashboard is running locally, and fetching a hystrix stream deployed to cloud foundry runtime. When accessing from the dashboard deployed to CF, hit the same connection timeout issue.

C-Otto commented 8 years ago

This is still an issue with 1.5.3 with both services running on Cloud Foundry.

jubinjacobthomas commented 7 years ago

i am also facing the same problem. It works fine when the app and dashboard are running locally. But when i deploy it to cloud foundry (i am using bluemix) it fails. No metrics are displayed. At times (that is after 2-3 restart of the app) i can see the metrics but it never gets refreshed.

mattrjacobs commented 7 years ago

I'm just not familiar enough with Cloud Foundry to get to the bottom of this. Anyone in the community willing to take a shot at it?

mukteshkrmishra commented 7 years ago

@jubinjacobthomas : Is there any error you are getting in CF logs?

jubinjacobthomas commented 7 years ago

Hystrix Dashboard logs on CF is as below.

2016-10-19T10:11:51.922+0530[App/0]OUTProxy opening connection to: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:11:51.924+0530[App/0]OUT2016-10-19 04:41:51.923 INFO 29 --- [io-61610-exec-3] ashboardConfiguration$ProxyStreamServlet : 2016-10-19T10:11:51.924+0530[App/0]OUTProxy opening connection to: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:12:02.431+0530[App/0]OUT at org.springframework.cloud.netflix.hystrix.dashboard.HystrixDashboardConfiguration$ProxyStreamServlet.doGet(HystrixDashboardConfiguration.java:172) ~[spring-cloud-netflix-hystrix-dashboard-1.2.0.RELEASE.jar!/:1.2.0.RELEASE] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:271) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT2016-10-19 04:42:02.429 ERROR 29 --- [io-61610-exec-2] ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:12:02.431+0530[App/0]OUTjava.net.SocketTimeoutException: Read timed out 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:82) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at java.net.SocketInputStream.socketRead(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:12:02.431+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.431+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at java.lang.Thread.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.432+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.435+0530[App/0]OUT at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0-internal] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:271) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.435+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:12:02.435+0530[App/0]OUTjava.net.SocketTimeoutException: Read timed out 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:82) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.435+0530[App/0]OUT2016-10-19 04:42:02.429 ERROR 29 --- [io-61610-exec-3] ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.435+0530[App/0]OUT at java.net.SocketInputStream.socketRead(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.435+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:158) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.cloud.netflix.hystrix.dashboard.HystrixDashboardConfiguration$ProxyStreamServlet.doGet(HystrixDashboardConfiguration.java:172) ~[spring-cloud-netflix-hystrix-dashboard-1.2.0.RELEASE.jar!/:1.2.0.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:12:02.436+0530[App/0]OUT at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:12:02.437+0530[App/0]OUT at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:07.983+0530[App/0]OUTProxy opening connection to: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:13:07.983+0530[App/0]OUT2016-10-19 04:43:07.982 INFO 29 --- [io-61610-exec-5] ashboardConfiguration$ProxyStreamServlet : 2016-10-19T10:13:07.983+0530[App/0]OUT2016-10-19 04:43:07.982 INFO 29 --- [io-61610-exec-6] ashboardConfiguration$ProxyStreamServlet : 2016-10-19T10:13:07.983+0530[App/0]OUTProxy opening connection to: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:271) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.springframework.cloud.netflix.hystrix.dashboard.HystrixDashboardConfiguration$ProxyStreamServlet.doGet(HystrixDashboardConfiguration.java:172) ~[spring-cloud-netflix-hystrix-dashboard-1.2.0.RELEASE.jar!/:1.2.0.RELEASE] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:158) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at java.net.SocketInputStream.socketRead(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:13:18.002+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:82) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.001+0530[App/0]OUTjava.net.SocketTimeoutException: Read timed out 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.001+0530[App/0]OUT2016-10-19 04:43:18.000 ERROR 29 --- [io-61610-exec-6] ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:13:18.004+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:13:18.002+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT2016-10-19 04:43:18.002 ERROR 29 --- [io-61610-exec-5] ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://hystrixappspring.mybluemix.net/hystrix.stream 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at java.lang.Thread.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUTjava.net.SocketTimeoutException: Read timed out 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:271) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:82) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.004+0530[App/0]OUT at java.net.SocketInputStream.socketRead(Unknown Source) ~[na:1.8.0-internal] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.003+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.002+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:158) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.002+0530[App/0]OUT at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.5.jar!/:4.4.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.springframework.cloud.netflix.hystrix.dashboard.HystrixDashboardConfiguration$ProxyStreamServlet.doGet(HystrixDashboardConfiguration.java:172) ~[spring-cloud-netflix-hystrix-dashboard-1.2.0.RELEASE.jar!/:1.2.0.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.003+0530[App/0]OUT at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.004+0530[App/0]OUT at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.004+0530[App/0]OUT at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar!/:4.5.2] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.3.RELEASE.jar!/:4.3.3.RELEASE] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at java.lang.Thread.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0-internal] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.5.jar!/:8.5.5] 2016-10-19T10:13:18.005+0530[App/0]OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0-internal]

jubinjacobthomas commented 7 years ago

@mukteshkrmishra The logs are posted above. The apps works fine locally. Also I am able to see the stream if i hit from chrome browser. My issue is similar to the issue mentioned in this link (http://stackoverflow.com/questions/37551950/not-able-to-get-the-metric-charts-displayed-using-hystrix-dashboard-in-bluemix )

jubinjacobthomas commented 7 years ago

Hi, i believe the issue is more to do with bluemix. I pushed the same apps to pivotal cloud foundry and it works there.

gangchen03 commented 7 years ago

The metrics streaming from Hystrix server to Hystrix dash assumes a long running HTTP connection, whereby server side keeps flushing down metrics data using SSE format. Cloud foundry does not support long running HTTP connections, so instead we used web sockets for streaming stuff. One of our team members created a web socket implementation equivalent to Hystrix metrics servlet and now we can stream Hystrix metrics from Bluemix without issues.

jubinjacobthomas commented 7 years ago

@gangchen03 Will you be able to share some sample code for the same ?

gangchen03 commented 7 years ago

Will share the code shortly in a sample git project.

gangchen03 commented 7 years ago

here is a workaround https://github.com/dunchych/hystrix-dashboard-bluemix

jubinjacobthomas commented 7 years ago

@gangchen03 Thanks for sharing the code. I will try this and let you know.

mattrjacobs commented 7 years ago

Closing due to inactivity. Please re-open if there's more to discuss.

C-Otto commented 7 years ago

@mattrjacobs as far as I'm concerned, this is still an issue worth fixing. I don't see a good reason to close this.

mattrjacobs commented 7 years ago

Reopening in that case for further discussion

YahyaHARRATHI commented 7 years ago

I had the same problem and it was my URL -_- I put " URL http://localhost:8080/hystrix.stream" so when changing that to "http://localhost:8080/hystrix.stream" it worked

jwconway commented 7 years ago

@mattrjacobs we're seeing a similar issue trying to run turbine and hystrix.streams inside of containers.

Locally everything works fine and the hystrix dashboard processes the stream from our zuul reverse proxy. We use turbine to aggregate the streams from all instances of the zuul instances. Again this works fine locally. When we try to deploy turbine to a container though, we see nothing form the turbine stream and hsytrix dashboard just hangs in "Loading...". Looking at the turbine server it does process connections from the client:

2017-01-25 13:36:19.529  INFO 1 --- [o-eventloop-3-1] o.s.c.n.t.s.TurbineStreamConfiguration   : SSE Request Received
2017-01-25 13:36:19.547  INFO 1 --- [o-eventloop-3-1] o.s.c.n.t.s.TurbineStreamConfiguration   : Starting aggregation
2017-01-25 13:36:50.558  INFO 1 --- [o-eventloop-3-1] o.s.c.n.t.s.TurbineStreamConfiguration   : Unsubscribing RxNetty serve
2017-01-25 13:36:50.559  INFO 1 --- [o-eventloop-3-1] o.s.c.n.t.s.TurbineStreamConfiguration   : Unsubscribing aggregation.

Interestingly it works fine if i have fiddler open... :/

gs-offcl commented 6 years ago

Dear Spring Cloud Team, I have a spring boot application with @enableturbinestream and @enablehystrixdashboard enabled.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-turbine-stream</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>

I have jersey restful services with following configurations added in pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-bus-amqp</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-netflix-hystrix-stream</artifactId>
</dependency>

and, I have rabbitmq also installed.

My clarification here is --

Does hystrix dash board/turbine supports for jersey apis endpoints to capture metrics ?

Currently i'm unable to see any captured metrics stream on Hystrix dashboard.

Any help would really appreciated Thank you.

GS

vspiliopoulos commented 5 years ago

@C-Otto @mattrjacobs I am having the same issue as well: After five calls to hystrix.stream (ONLY via an AWS load balancer - via localhost it works fine) all connections are consumed and the sixth call fails with http 503. I took a closer look and it seems that all connections created on the microservice that runs hystix.stream (client -> AWS LB -> microservice) stay forever in state ESTABLISHED.

[root@ip- blabla]# netstat -anop | grep 8080 | grep EST tcp6 0 0 X.X.X.X:8080 X.X.X.X:21768 ESTABLISHED 18077/java off (0.00/0/0)

This is true even if the client kills the connection. My assumption is that the aws LB is keeping the connection to the service and because hystrix data are flowing it never times out. This sounds to me like an AWS LB configuration issue, but I am not sure. Any thoughts on this?