search

fabric8io / kubeflix

Kubernetes integration with Netflix OSS
266 stars 69 forks source link

Unable to connect to Command Metric Stream. (with Kubeflix 1.0.16 and 1.0.17) #90

Closed rafabene closed 8 years ago

rafabene commented 8 years ago

When I deployed Kubeflix 1.0.16 that uses openzipkin/zipkin-query:1.39.3 image I just see the following messages when I open the hystrix-dashboard:

In the Turbine Pod log I see a lot of exceptions:

016-05-10 15:41:33.481 ERROR [turbine-server,,,] 1 --- [InstanceMonitor] c.n.t.monitor.instance.InstanceMonitor   : Could not initiate connection to host, giving up: [Cannot GET /hystrix.stream]
480 2016-05-10 15:41:33.482  WARN [turbine-server,,,] 1 --- [InstanceMonitor] c.n.t.monitor.instance.InstanceMonitor   : Stopping InstanceMonitor for: 172.17.0.5 helloworld-msa
481 
482 com.netflix.turbine.monitor.instance.InstanceMonitor$MisconfiguredHostException: [Cannot GET /hystrix.stream]
483     at com.netflix.turbine.monitor.instance.InstanceMonitor.init(InstanceMonitor.java:318) ~[turbine-core-1.0.0.jar!/:na]
484     at com.netflix.turbine.monitor.instance.InstanceMonitor.access$100(InstanceMonitor.java:103) ~[turbine-core-1.0.0.jar!/:na]
485     at com.netflix.turbine.monitor.instance.InstanceMonitor$2.call(InstanceMonitor.java:235) [turbine-core-1.0.0.jar!/:na]
486     at com.netflix.turbine.monitor.instance.InstanceMonitor$2.call(InstanceMonitor.java:229) [turbine-core-1.0.0.jar!/:na]
487     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
488     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
489     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
490     at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
iocanel commented 8 years ago

I think that 1.0.16 had a couple of issues that have already been fixed in the context of #78.

We should have a new release with the fixes soon

rafabene commented 8 years ago

The issue still happening with 1.0.17

iocanel commented 8 years ago

@rafabene: sorry to hear that.

It seems that instances are discovered, but for some reason turbine can't "see" the metrics stream. I guess that I could possibly try to reproduce that issue by using the steps described here https://github.com/redhat-helloworld-msa/helloworld-msa, right?

rafabene commented 8 years ago

Exactly. I'm using https://github.com/redhat-helloworld-msa/helloworld-msa as my test environment for Kubeflix

iocanel commented 8 years ago

Ok, I am already on it. I guess I should try to test future releases against that or even write some system tests, so that we avoid regressions.

rafabene commented 8 years ago

Thanks. That would be great

On Fri, May 13, 2016 at 11:50 AM, Ioannis Canellos <notifications@github.com

wrote:

Ok, I am already on it. I guess I should try to test future releases against that or even write some system tests, so that we avoid regressions.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/fabric8io/kubeflix/issues/90#issuecomment-219083142

Rafael Benevides | Senior Product Manager Director of Developer Experience M: +1-919-592-6255

Better technology. Faster innovation. Powered by community collaboration. See how it works at www.redhat.com

iocanel commented 8 years ago

On a clean vagrant environment (using the fabric8 vagrant image) I used the following steps:

https://github.com/redhat-helloworld-msa/helloworld-msa/blob/master/create-project.adoc https://github.com/redhat-helloworld-msa/helloworld-msa/blob/master/kubeflix.adoc https://github.com/redhat-helloworld-msa/helloworld-msa/blob/master/ola.adoc https://github.com/redhat-helloworld-msa/helloworld-msa/blob/master/hola.adoc

I am sending requests to http://ola-helloworld-msa.vagrant.f8/

and I receive back:

[
  "Olá de ola-tgkyp",
  "Hola de hola-7nofr",
  "Aloha response (fallback)"
]

Which seems correct to me. Also the hystrix dashboard and the zipkin console seem to work fine.

Some my guess is, that in your case turbine fails to access the hystrix stream of one of the services I was too lazy (e.g. hello, aloha, namaste) to try and it barfs.

I will try to add the remaining services one by one to see if I can spot something there....

iocanel commented 8 years ago

@rafabene: Do you still have issues with this? I didn't have much luck reproducing it.

rafabene commented 8 years ago

Sorry @iocanel but It seems that it's working now. Everything perfect with 1.0.17

iocanel commented 8 years ago

@rafabene: np! I am really glad that its working again :-)