search

Netflix / Turbine

SSE Stream Aggregator
Apache License 2.0
836 stars 255 forks source link

Turbine is unable to aggregate the response from clusters and write it back to response stream. #29

Open rvangapal opened 10 years ago

rvangapal commented 10 years ago

Hello,

I am planning to leverage hystrix dashboard and turbine which is very useful for us.

I am trying to configure turbine to aggregate metrics from clustered hystrix service apps. For some reason, it's not working. I am not sure whether my config is wrong or is it a bug. As cluster config is failing, i tried to configure single node config. Even that is also failing. I hope i have provided required configuration for you review, hope you can guide me on how to address this. Logs are captured from catalina.out.

CONFIGURATION

Platform: RHEL 6.3 Tomcat version: 6.0.28 Turbine version: turbine-web-1.0.0.war (downloaded from hystrix wiki)

Cluster config:

config.properties

turbine.aggregator.clusterConfig=dev-lsds turbine.FileBasedInstanceDiscovery.filePath=/opt/usr/apps/tomcat-6.0.28/webapps/turbine/WEB-INF/classes/hosts.csv turbine.instanceUrlSuffix=:1776/hystrix.stream InstanceDiscovery.impl=com.netflix.turbine.discovery.FileBasedInstanceDiscovery

hosts.csv:

dev-lsds-1.tnt13-zone1.aus1,dev-lsds,up

Single node config:

config.properties

turbine.ConfigPropertyBasedDiscovery.default.instances=dev-lsds-1.tnt13-zone1.aus1 turbine.instanceUrlSuffix=:1776/hystrix.stream

OBSERVATIONS

Hystrix node (SUCCESS):

URL: http://dev-lsds-1.tnt13-zone1.aus1:1776/hystrix.stream Response: It is returning successful json response

Turbine node (FAILS)

**** SINGLE NODE CONFIG *****

URL: http://localhost:8080/turbine/turbine.stream

Response :

curl -v http://localhost:8080/turbine/turbine.stream

Here is the log output for this request. Log: http://pastebin.com/ywUjHCz9

**** CLUSTER NODE CONFIG *****

URL: http://localhost:8080/turbine/turbine.stream?cluster=dev-lsds RESPONSE: curl -v http://localhost:8080/turbine/turbine.stream?cluster=dev-lsds

LOG1: (Same log out put as single node confg ) http://pastebin.com/ywUjHCz9

LOG2: (After couple of attempts, reached 503 max connection requests) http://pastebin.com/PmUWH52v

Please let me know if you need more information.

rvangapal commented 10 years ago

Hello turbine team,

When can you get a chance, can you please look into this and reply this?

Thanks, Ravi

scott-seo commented 10 years ago

I had trouble with the downloaded version of turbine-web-1.0.0.war. I was seeing similar response from turbine as you. I read this post. https://github.com/Netflix/Turbine/issues/13

I built my own web app and then embedded turbine into it. that worked.

I also tried building war from Turbine/turbine-web project. that worked.

I had to make sure two things are set up correctly. first config.properties.

InstanceDiscovery.impl=com.netflix.turbine.discovery.ConfigPropertyBasedDiscovery
turbine.aggregator.clusterConfig=default
turbine.instanceUrlSuffix=:9090/test-app/hystrix.stream
turbine.ConfigPropertyBasedDiscovery.default.instances=X.X.X.X  <== your host ip address or hostname

Next, I had to make sure my webapp that's executing the Hystrix commands are actively generating hystrix stream. Otherwise, you won't see much in Hystrix dashboard.

Give this a try.

benjchristensen commented 10 years ago

What is wrong with turbine-web-1.0.0.war?

--  Ben Christensen - Netflix Edge Engineering +1.310.782.5511  @benjchristensen

On September 14, 2014 at 7:43:20 AM, Scott Seo (notifications@github.com) wrote:

I had trouble with the downloaded version of turbine-web-1.0.0.war. I was seeing similar response from turbine as you. I read this post. #13

I built my own web app and then embedded turbine into it. that worked.

I also tried building war from Turbine/turbine-web project. that worked.

I had to make sure two things are set up correctly. first config.properties.

InstanceDiscovery.impl=com.netflix.turbine.discovery.ConfigPropertyBasedDiscovery turbine.aggregator.clusterConfig=default turbine.instanceUrlSuffix=:9090/test-app/hystrix.stream turbine.ConfigPropertyBasedDiscovery.default.instances=X.X.X.X <== your host ip address or hostname

Next, I had to make sure my webapp that's executing the Hystrix commands are actively generating hystrix stream. Otherwise, you won't see much in Hystrix dashboard.

Give this a try.

— Reply to this email directly or view it on GitHub.

scott-seo commented 10 years ago 
 Sorry it's hard to tell what was exactly wrong with my environment at

that point. It's most likely my environment was wrong. All I was seeing was pings like the person who filed this original post. I was not seeing [data] responses from Turbine. BTW, I was connecting from my local environment to AWS. The latency was over 3.5 seconds. Debug statement did say something about skipping to catch up with the stream.

At this point I have decided to go with recommendation made by one of

your engineers on post #13.

On Mon, Sep 15, 2014 at 12:11 PM, Ben Christensen notifications@github.com wrote:

What is wrong with turbine-web-1.0.0.war?

Ben Christensen - Netflix Edge Engineering +1.310.782.5511 @benjchristensen

On September 14, 2014 at 7:43:20 AM, Scott Seo (notifications@github.com) wrote:

I had trouble with the downloaded version of turbine-web-1.0.0.war. I was seeing similar response from turbine as you. I read this post. #13

I built my own web app and then embedded turbine into it. that worked.

I also tried building war from Turbine/turbine-web project. that worked.

I had to make sure two things are set up correctly. first config.properties.

InstanceDiscovery.impl=com.netflix.turbine.discovery.ConfigPropertyBasedDiscovery

turbine.aggregator.clusterConfig=default turbine.instanceUrlSuffix=:9090/test-app/hystrix.stream turbine.ConfigPropertyBasedDiscovery.default.instances=X.X.X.X <== your host ip address or hostname

Next, I had to make sure my webapp that's executing the Hystrix commands are actively generating hystrix stream. Otherwise, you won't see much in Hystrix dashboard.

Give this a try.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/Netflix/Turbine/issues/29#issuecomment-55614044.

karthik101 commented 9 years ago

@scott-seo I am trying to build turbine-web war but after building i cant find the created war. confused on to how build this web app.

mohan-mishra commented 8 years ago

@scott-seo I too am unable to make the war. Could you please help