scylladb / java-driver

ScyllaDB Java Driver for ScyllaDB and Apache Cassandra, based on the DataStax Java Driver
Apache License 2.0
60 stars 36 forks source link

java.lang.NullPointerException: Cannot read the array length because "<local2>" is null #85

Open dkropachev opened 3 years ago

dkropachev commented 3 years ago

Faced this issue once while playing with nosqlbench:

[cluster1-worker-0] ERROR com.datastax.cql3.scylla_shaded.driver.core.ExceptionCatchingRunnable - Unexpected error while executing task
java.lang.NullPointerException: Cannot read the array length because "<local2>" is null
    at com.datastax.cql3.scylla_shaded.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:841)
    at com.datastax.cql3.scylla_shaded.driver.core.SessionManager.removePool(SessionManager.java:440)
    at com.datastax.cql3.scylla_shaded.driver.core.SessionManager.onDown(SessionManager.java:528)
    at com.datastax.cql3.scylla_shaded.driver.core.Cluster$Manager.onDown(Cluster.java:2207)
    at com.datastax.cql3.scylla_shaded.driver.core.Cluster$Manager.access$1200(Cluster.java:1560)
    at com.datastax.cql3.scylla_shaded.driver.core.Cluster$Manager$5.runMayThrow(Cluster.java:2162)
    at com.datastax.cql3.scylla_shaded.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at com.datastax.internal.com_google_common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
    at com.datastax.internal.com_google_common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
    at com.datastax.internal.com_google_common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:832)
[cqltabular2_default_schema:001] WARN com.datastax.cql3.scylla_shaded.driver.core.Session - Error creating pool to /10.0.2.65:9042
com.datastax.cql3.scylla_shaded.driver.core.exceptions.OperationTimedOutException: [/10.0.2.65:9042] Operation timed out
    at com.datastax.cql3.scylla_shaded.driver.core.Connection$Future.onTimeout(Connection.java:1642)
    at com.datastax.cql3.scylla_shaded.driver.core.Connection$ResponseHandler$1.run(Connection.java:1721)
    at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
    at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
    at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:832)
dkropachev commented 3 years ago

Problem goes from src/main/java/com/datastax/driver/core/HostConnectionPool.java:

We have 5 attributes that needs to be initialized before being used:

  List<Connection>[] connections;
  private AtomicInteger[] open;
  @VisibleForTesting Set<Connection>[] trash;
  private Queue<PendingBorrow>[] pendingBorrows;
  private AtomicInteger[] scheduledForCreation;

But we initialize them only at HostConnectionPool.initAsyncWithConnection , which may not happen due to the exception escaping at HostConnectionPool.initAsync:

  ListenableFuture<Void> initAsync(Connection reusedConnection) {
    if (reusedConnection != null && reusedConnection.setOwner(this)) {
      return initAsyncWithConnection(reusedConnection);
    }
    try {
      return initAsyncWithConnection(manager.connectionFactory().open(this));
    } catch (Exception e) {
      phase.compareAndSet(Phase.INITIALIZING, Phase.INIT_FAILED);
      SettableFuture<Void> future = SettableFuture.create();
      future.setException(e);
      return future;
    }
  }

So basically when initial connection fails we end up with perfectly uninitialized instance