search

apache / drill

Apache Drill is a distributed MPP query layer for self describing data
https://drill.apache.org/
Apache License 2.0
1.95k stars 979 forks source link

Apache Cassandra query always return No node was available to execute the query #2902

Open ijbgreen opened 7 months ago

ijbgreen commented 7 months ago

Describe the bug I have an Apache Cassandra cluster configured with three nodes. To which I need to connect and execute queries. But when I perform any type of Drill operation it always returns the following error: com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query. I have tried to change the topology of the Cassandra cluster thinking that would be the problem but no. This error does not occur with other connectors that we use, like python modules that we use to process data to Cassandra cluster.

To Reproduce 1.-Configure de Cassandra Storage Plugin 2.-Try to query any table in any keyspace

Expected behavior Query data return

Error detail, log output or screenshots Open failed for reader: EnumerableRecordReader Fragment: 0:0

[Error Id: 308d678c-b8e7-4377-8b32-507460a43a93 on ip-10-125-112-16.ec2.internal:31010]

(com.datastax.oss.driver.api.core.NoNodeAvailableException) No node was available to execute the query com.datastax.oss.driver.api.core.NoNodeAvailableException.copy():40 com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly():149 com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process():53 com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process():30 com.datastax.oss.driver.internal.core.session.DefaultSession.execute():237 com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute():54 com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute():78 org.apache.calcite.adapter.cassandra.CassandraTable$2.enumerator():223 org.apache.calcite.linq4j.AbstractEnumerable.iterator():33 java.lang.Iterable.spliterator():101 org.apache.drill.exec.store.enumerable.EnumerableRecordReader.setup():107 org.apache.drill.exec.store.enumerable.EnumerableRecordReader.open():146 org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.open():211 org.apache.drill.exec.physical.impl.scan.framework.ShimBatchReader.open():76 org.apache.drill.exec.physical.impl.scan.ReaderState.open():224 org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.nextAction():286 org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.next():242 org.apache.drill.exec.physical.impl.protocol.OperatorDriver.doNext():201 org.apache.drill.exec.physical.impl.protocol.OperatorDriver.start():179 org.apache.drill.exec.physical.impl.protocol.OperatorDriver.next():129 org.apache.drill.exec.physical.impl.protocol.OperatorRecordBatch.next():149 org.apache.drill.exec.record.AbstractRecordBatch.next():109 org.apache.drill.exec.record.AbstractRecordBatch.next():101 org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():59 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():93 org.apache.drill.exec.record.AbstractRecordBatch.next():160 org.apache.drill.exec.physical.impl.BaseRootExec.next():103 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81 org.apache.drill.exec.physical.impl.BaseRootExec.next():93 org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0():321 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():423 org.apache.hadoop.security.UserGroupInformation.doAs():1762 org.apache.drill.exec.work.fragment.FragmentExecutor.run():310 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1128 java.util.concurrent.ThreadPoolExecutor$Worker.run():628 java.lang.Thread.run():834

at org.apache.drill.exec.server.rest.RestQueryRunner.submitQuery(RestQueryRunner.java:99)
at org.apache.drill.exec.server.rest.RestQueryRunner.run(RestQueryRunner.java:54)
at org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:161)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
at org.apache.drill.exec.server.rest.header.ResponseHeadersSettingFilter.doFilter(ResponseHeadersSettingFilter.java:71)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.drill.exec.server.rest.CsrfTokenValidateFilter.doFilter(CsrfTokenValidateFilter.java:55)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.apache.drill.exec.server.rest.CsrfTokenInjectFilter.doFilter(CsrfTokenInjectFilter.java:54)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571)
at org.apache.drill.exec.server.rest.auth.DrillHttpSecurityHandlerProvider.handle(DrillHttpSecurityHandlerProvider.java:163)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.lang.Exception: No node was available to execute the query at com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40) at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149) at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53) at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30) at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:237) at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54) at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:78) at org.apache.calcite.adapter.cassandra.CassandraTable$2.enumerator(CassandraTable.java:223) at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33) at .......(:0) at org.apache.drill.exec.store.enumerable.EnumerableRecordReader.setup(EnumerableRecordReader.java:107) at org.apache.drill.exec.store.enumerable.EnumerableRecordReader.open(EnumerableRecordReader.java:146) at org.apache.drill.exec.physical.impl.scan.framework.ManagedScanFramework.open(ManagedScanFramework.java:211) at org.apache.drill.exec.physical.impl.scan.framework.ShimBatchReader.open(ShimBatchReader.java:76) at org.apache.drill.exec.physical.impl.scan.ReaderState.open(ReaderState.java:224) at org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.nextAction(ScanOperatorExec.java:286) at org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.next(ScanOperatorExec.java:242) at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.doNext(OperatorDriver.java:201) at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.start(OperatorDriver.java:179) at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.next(OperatorDriver.java:129) at org.apache.drill.exec.physical.impl.protocol.OperatorRecordBatch.next(OperatorRecordBatch.java:149) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:101) at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:59) at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:93) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:160) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103) at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) at org.apache.drill.exec.work.fragment.FragmentExecutor.lambda$run$0(FragmentExecutor.java:321) at .......(:0) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:310) at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) at .......(:0) 2024-04-18 22:43:55,631 [19de0973-8750-a353-e6ab-6c091311b934:frag:0:0] WARN o.a.d.e.w.f.QueryStateProcessor - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal).

Drill version Drill Version 1.21.1

Additional context In many forums i read that this issue is because the query doesn´t have a datacenter defined in the Cassandra Connection. But i can't find where in the configuration use it. In fact we only can configure one node of the Cassandra Cluster cassandra.json drillbit.log