Unable to connect to Command Metric Stream

[AshutoshSharma10]

Hi,

We are using the Spring boot starter parent 1.5.2.RELEASE and hystrix 1.5.6 dependencies. Fortunately it works once in hundred tries. We are really puzzled, why it works once and then never. All the services are running on the local machine. The log details are as below: Proxy opening connection to: http://localhost:6001/hystrix.stream 2017-05-05 12:29:16.951 INFO 4876 --- [nio-6001-exec-5] ashboardConfiguration$ProxyStreamServlet : Proxy opening connection to: http://localhost:6001/hystrix.stream 2017-05-05 12:31:36.163 INFO 4876 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration

We are using the actuator also and tried the management.port once but the result is same. So we removed it. The same thing we tried with turbine and it also behaves the same way.

Please suggest.

Below is the resulting screen:

[mohanaraosv]

@AshutoshSharma10 Could you please create sample project which simulates the above problem. Is the server polling the data?

[mattrjacobs]

@AshutoshSharma10 It's very difficult to provide an answer with those details. One possible way of getting to the bottom of it would be for you to look at the sample application that properly serves the metrics stream, and find out where yours differs.

https://github.com/Netflix/Hystrix/tree/master/hystrix-examples-webapp

[michelbetancourt]

I want to plus this issue -- we are seeing the same issue with Spring-boot 1.4.x and not just 1.5.x.

If we try to curl the end point instead of getting back a snapshot of data, we get

Previous versions of Spring-boot 1.3.x did not have an issue. At first sight, this seems to be an incompatibility with Spring-boot or one of it's dependencies since our team had not had issues until a recent upgrade with Hystrix.

Noteworthy comment is that all other parts of Hystrix are working just fine. RequestLog, and at least Circuit Breaking are working as designed. For some reason the Hystrix Streaming Servlet is no longer functional for us.

Is there any way to troubleshoot this issue?

The symptoms we receive for this issue are as follows:

• Unable to connect to Command Stream (per description of ticket)
• Access log receives 503 http status entry for hystrix.stream end point
• The curl below hangs:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /debug/hystrix.stream HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.51.0
> Accept: */*
>```

[michelbetancourt]

If time permits we will try to put together a sandbox sample app, if one is not already provided.

[michelbetancourt]

Found the issue on our end. We added a delay in the writing of the response so that we can propagate headers in our response the way we needed them. Hystrix Servlet does not play well with our Response-delaying mechanism.

[615Manu]

Hello,

We are also having a spring boot app and trying to see the metrics on the hystrix dashboard, But getting exactly the same issue. Is there any update on this issue. We are using hystrix 1.5.3. Any fix or workaround would be helpful Thanks in Advance

[mattrjacobs]

@615Manu The original issue was resolved. Every circumstance is potentially different, and you haven't listed any details about your problem.

I would start by looking at the sample application that properly serves the metrics stream, and find out where yours differs.

https://github.com/Netflix/Hystrix/tree/master/hystrix-examples-webapp

[asgs]

@615Manu @AshutoshSharma10 For spring boot apps to show Hystrix metrics, the spring starter spring-boot-starter-actuator should be added to your dependency. This will apparently enable the metrics filter.

Also, make sure you open the http://<host>:<port>/hystrix/monitor URL and type http://<host>:<port>/hystrix.stream in the textbox. Then hit the Monitor Stream button to see the live stats.

[sunday2]

i also encountered the same problem ,and the version is 1.4.3

[Credo-Zhao]

I have the same question...version is 2.0

[tomfrans]

management.endpoints.web.exposure.include=* adding in application.properties solved for me

[spencergibb]

This isn't the spring cloud Netflix repository

[ezhong0812]

same question , version 2.0.0 ,
http://localhost:8764/hystrix.stream get the same page with http://localhost:8764/hystrix

[spencergibb]

This isn't the spring cloud Netflix repository

[cion-J]

version 2.0.0. "management.endpoints.web.exposure.include=hystrix.stream" adding in bootstrap.properties; and the url was change to "http://host:port/actuator/hystrix.stream", it is ok. also,"management.endpoints.web.base-path=/ " can make "http://host:port/hystrix.stream" work.

[moviewang]

@cion-J Yours answer solved for me! thks!

[ww530220264]

hi,i just occured this problem,after one day, i may have solved this,please add a class in your project

version:2.0

package com.sc.consumerribbon.config;

import com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet; import org.springframework.boot.web.servlet.ServletRegistrationBean; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration;

@Configuration public class HystrixServletDefinitions {

@Bean(name = "hystrixRegistrationBean")
public ServletRegistrationBean servletRegistrationBean() {
    ServletRegistrationBean registration = new ServletRegistrationBean(
            new HystrixMetricsStreamServlet(), "/hystrix.stream");
    registration.setName("hystrixServlet");
    registration.setLoadOnStartup(1);
    return registration;
}

} @AshutoshSharma10 可以看下我记录的 https://blog.csdn.net/qq_22772259/article/details/79812026

[1029574707]

@ww530220264 Thanks guy, I have solved this problem in my project with your method.

[playerlilin]

@ww530220264 Thanks for a lot,your method is great.

[rajeshrengarajulu]

The above solution is for 2.0.x. However it's still not working with Spring-boot 1.4.x and 1.5.x. Any updates on that?

[spencergibb]

This isn't the spring cloud Netflix repository

[duanhaowei]

this solution is great , so may be is a miss part of cloud document? thanks @ww530220264

[kennyk65]

Solution from @cion-J is great! Spring Cloud docs require minor adjustment to reflect the change. My suggested PR: https://github.com/kennyk65/spring-cloud-netflix/pull/1

[ghost]

spring cloud doc is really sucks...

[lennyxue]

just added one class as below : @Configuration public class HystrixServletDefinitions { @Bean(name = "hystrixRegistrationBean") public ServletRegistrationBean servletRegistrationBean() { ServletRegistrationBean registration = new ServletRegistrationBean( new HystrixMetricsStreamServlet(), "/metrics/hystrix.stream"); registration.setName("hystrixServlet"); registration.setLoadOnStartup(1); return registration; } }

issues resolved.

[MrSummer33]

if there was no HystrixCommand data , the hystrix server log "java.net.SocketTimeoutException: Read timed out" . it also show you "Unable to connect to Command Metric Stream" on the dashboard .
so you can request the HystrixCommad .then it will be ok the detail info in HystrixSampleSseServlet.handleRequest()

[romankh3]

version 2.0.0. "management.endpoints.web.exposure.include=hystrix.stream" adding in bootstrap.properties; and the url was change to "http://host:port/actuator/hystrix.stream", it is ok. also,"management.endpoints.web.base-path=/ " can make "http://host:port/hystrix.stream" work.

I have the same problem and the solution above works for me! Look very strange and not user-friendly...

[BroniusJanonis]

For me on Spring-Boot version 2 helped with dependencies

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

Added anotation to aplication @EnableHystrixDashboard Added bootstrap poperties (not sure why in some examples they are not using or why are they using here)

management:
  endpoints:
    web:
      exposure:
        include: hystrix.stream
      base-path: /

Run the app Opened first http://host:port/hystrix then added in the url http://host:port/hystrix.stream and it worked If I used http://host:port/actuator/hystrix.stream I was getting errors

[rajanikanthbhagavan]

version 2.0.0. "management.endpoints.web.exposure.include=hystrix.stream" adding in bootstrap.properties; and the url was change to "http://host:port/actuator/hystrix.stream", it is ok. also,"management.endpoints.web.base-path=/ " can make "http://host:port/hystrix.stream" work.

This worked for me, thank you!!

[Sicmatr1x]

You should add both @EnableHystrixDashboard and @EnableCircuitBreaker

like this:

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
@EnableHystrixDashboard
@EnableCircuitBreaker
public class ConsumerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConsumerApplication.class, args);
    }
}

[duansiman]

http://localhost:59001/actuator/hystrix.stream Fill in the link here and change the port to your administrative port. note the 'actuator'

[hxd890724]

hi,i just occured this problem,after one day, i may have solved this,please add a class in your project

version:2.0

package com.sc.consumerribbon.config;

import com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet; import org.springframework.boot.web.servlet.ServletRegistrationBean; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration;

@configuration public class HystrixServletDefinitions {

@Bean(name = "hystrixRegistrationBean")
public ServletRegistrationBean servletRegistrationBean() {
    ServletRegistrationBean registration = new ServletRegistrationBean(
            new HystrixMetricsStreamServlet(), "/hystrix.stream");
    registration.setName("hystrixServlet");
    registration.setLoadOnStartup(1);
    return registration;
}

} @AshutoshSharma10 可以看下我记录的 https://blog.csdn.net/qq_22772259/article/details/79812026

Solved

[Rtwkkumar]

not working.Please help :(

Spring version

`

org.springframework.boot spring-boot-starter-parent 1.5.9.RELEASE

`

[Vishalckc]

Facing same issue in spring version 2.3.3. I have two services and two client balanced services consuming these other two services. A separate service for visualizing the Eureka Dashboard with below properties. But still same error

main class annotated @EnableHystrixDashboard#

Configuration class added as suggested in the above discussions

import com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet;
import org.springframework.boot.web.servlet.ServletRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class HystrixServletDefinitions {

    @Bean(name = "hystrixRegistrationBean")
    public ServletRegistrationBean servletRegistrationBean() {
        ServletRegistrationBean registration = new ServletRegistrationBean(
                new HystrixMetricsStreamServlet(), "/hystrix.stream");
        registration.setName("hystrixServlet");
        registration.setLoadOnStartup(1);
        return registration;
    }
}

Pom.XML-

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.3.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>pluralsight.demo</groupId>
    <artifactId>pluralsight-hystrix-dashboard</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>pluralsight-hystrix-dashboard</name>
    <description>Demo of hystrix</description>

    <properties>
        <java.version>1.8</java.version>
        <spring-cloud.version>Hoxton.SR8</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

bootstrap.properties-

server.port=8085
eureka.client.register-with-eureka=false
management.endpoints.web.exposure.include=hystrix.stream
management.endpoints.web.base-path=/
management.security.enabled=false

[littiele]

Hello everyone, everything good ?

I implemented Hystrix however when I run localhost everything works fine but when I go up to the cloud (AWS) it loads however if I enter the url actuator / hystrix.stream I can see the ping normally. However, when I access the dashboard, the message "Unable to connect to Command Metric Stream." Someone has any idea what it might be because I have already implemented everything that is saying here but it will not.

[ashfaqnisar]

Were you able to resolve this issue @littiele and @Vishalckc ?

[hendisantika]

Still not solve my issue:

Here the sample repo: https://gitlab.com/microservice-samples/micro-services-spring-cloud-config/hystrix-circuit-breaker

[subbu05]

Same Issue I am seeing via Docker container

[6harat]

Facing a similar issue, steps followed:

Dashboard was setup using steps suggested here: https://github.com/Netflix-Skunkworks/hystrix-dashboard/wiki#run-via-docker

Hystrix stream was generated as per steps mentioned by @mattrjacobs here: https://github.com/Netflix/Hystrix/issues/1117#issuecomment-192499462

verified that i am getting the stream with ping and data: http://localhost:8989/hystrix-examples-webapp/hystrix.stream

but able to see following stack trace on dashboard: hystrix-strack-trace.txt

Probably hystrix dashboard setup might be at fault (will check shortly)

UPDATE (Solution): since I am setting up dashboard using the docker image, it will be resolving localhost to the container and not to my local machine.

To fix this issue, replace your stream url http://localhost:8989/hystrix-examples-webapp/hystrix.stream with http://host.docker.internal:8989/hystrix-examples-webapp/hystrix.stream when entering it in the dashboard. this should lead to dashboard getting generated correctly. (refer this: https://stackoverflow.com/a/43541732/6687477 if host.docker.internal does not work for you. you might be on old docker version)