search

intuit / wasabi

Wasabi A/B Testing service is an open source project that is no longer under active development or being supported
Apache License 2.0
1.14k stars 241 forks source link

Cassandra connectivity is challenging #179

Open djMax opened 7 years ago

djMax commented 7 years ago

The docs on this could be a lot better. I'm getting gobs of this error trying to use a 3 node cassandra cluster in Kubernetes on aws:

77006 [cluster6-nio-worker-0] DEBUG c.datastax.driver.core.Host.STATES - [cassandra-1.cassandra/10.2.42.10:9042] Connection[cassandra-1.cassandra/10.2.42.10:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
77023 [main] DEBUG c.d.driver.core.ControlConnection - [Control connection] Refreshing node list and token map
77064 [main] DEBUG c.d.driver.core.ControlConnection - [Control connection] Refreshing schema
77428 [main] DEBUG c.datastax.driver.core.Host.STATES - [Control connection] established to cassandra-1.cassandra/10.2.42.10:9042
77429 [main] INFO  c.d.d.c.p.DCAwareRoundRobinPolicy - Using provided data-center name 'datacenter1' for DCAwareRoundRobinPolicy
77430 [main] WARN  c.d.d.c.p.DCAwareRoundRobinPolicy - Some contact points don't match local data center. Local DC = datacenter1. Non-conforming contact points: cassandra-0.cassandra/10.2.101.2:9042 (DC1-Stage),cassandra-1.cassandra/10.2.42.10:9042 (DC1-Stage),cassandra-2.cassandra/10.2.14.10:9042 (DC1-Stage)
77431 [main] INFO  com.datastax.driver.core.Cluster - New Cassandra host cassandra-0.cassandra/10.2.101.2:9042 added
77464 [main] INFO  com.datastax.driver.core.Cluster - New Cassandra host cassandra-1.cassandra/10.2.42.10:9042 added
77465 [main] INFO  com.datastax.driver.core.Cluster - New Cassandra host cassandra-2.cassandra/10.2.14.10:9042 added
77467 [main] ERROR c.i.w.c.d.DefaultCassandraDriver - Exception occurred while connecting to the cluster...
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
    at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.Cluster.connect(Cluster.java:286) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.intuit.wasabi.cassandra.datastax.DefaultCassandraDriver.initialize(DefaultCassandraDriver.java:166) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.intuit.wasabi.cassandra.datastax.DefaultCassandraDriver.<init>(DefaultCassandraDriver.java:83) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.intuit.wasabi.cassandra.datastax.DefaultCassandraDriver$$FastClassByGuice$$f0ff6476.newInstance(<generated>) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:226) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.netflix.governator.guice.LifecycleInjector.createChildInjector(LifecycleInjector.java:327) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.netflix.governator.guice.LifecycleInjector.createInjector(LifecycleInjector.java:394) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.netflix.governator.guice.LifecycleInjector.createInjector(LifecycleInjector.java:348) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.intuit.autumn.service.ServiceManager.getInjector(ServiceManager.java:207) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.intuit.autumn.service.ServiceManager.start(ServiceManager.java:164) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.intuit.wasabi.Main.main(Main.java:60) [wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
    at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:207) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:273) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:111) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:136) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.AbstractSession.executeAsync(AbstractSession.java:72) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.Cluster$1.apply(Cluster.java:344) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.Cluster$1.apply(Cluster.java:341) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.Futures.transform(Futures.java:608) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.google.common.util.concurrent.Futures.transform(Futures.java:561) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:341) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    at com.datastax.driver.core.Cluster.connect(Cluster.java:284) ~[wasabi-main-1.0.20170210223759-development-all.jar:1.0.20170210223759]
    ... 38 common frames omitted
djMax commented 7 years ago

I have some suspicion this is a version problem. Should wasabi work with the latest protocol?

reachym commented 7 years ago

@djMax Were you able to get it working with 3 nodes cluster by any chance? I am trying the same thing and having issues. Also, could you please share how you specified 3 nodes in pom.xml if possible?

djMax commented 7 years ago

In the end, we gave up on wasabi and built our own. In large part because we're a postgres/elastic search shop and I didn't want to have to maintain mysql/cassandra just for this.

yoshprogrammer commented 6 years ago

I'm getting this error in Kubernetes on Azure as well.

Edit: I'm actually running into this issue running any dockerized version of the application that isn't installed via the shell scripts.

yoshprogrammer commented 6 years ago

Yeah, I've tried a ton of things here.

  1. Upgrading the drivers to 3.4.0
  2. Upgrading/redeploying Cassandra versions 2.1 & 2.2 to the cluster
  3. Tweaking configs to see if I can get a connection live

I'm still stuck as I can manually cqlsh from my mac to the cluster, and I can use cqlsh once accessed into a shell in the Cassandra instance but Wasabi is failing to connect. Likely going to have to look at other solutions but disappointed as I've been impressed with the offering.

mariusciziunas commented 6 years ago

Has anyone managed to solve this? I have the same issue with Cassandra 3.11 and drivers 3.5.0

mariusciziunas commented 6 years ago

Managed to solve this.

The reason behind is Cassandra driver built by Datastax and the way it is used in the Wasabi application. There is LoadBalancingPolicy set in DefaultCassandraDriver.java:104. DCAwareRoundRobinPolicy doesn't work with more than one node. If set to TokenAwarePolicy then it works with multiple nodes. "Works" here means that I am able to start application, create experiments manipulate data and real time APIs. Changing the policy might cause other issues for loadbalancing/cluster, I don't know yet...

Dracks commented 6 years ago

The real problem is that cassandra uses the cassandra DataCenter name, and if that doesn't match, it cannot connect, this is why it says 3 available, no hosts tried. Wasabi is tring to found hosts of cassandra with the specified DataCenter name.