search

strapdata / elassandra

Elassandra = Elasticsearch + Apache Cassandra
http://www.elassandra.io
Apache License 2.0
1.72k stars 199 forks source link

Unable to delete index after node has been removed from cluster. #381

Open filthz opened 3 years ago

filthz commented 3 years ago

Elassandra 6.2.3.23 I have removed a node from my cluster:

2020-12-14 07:02:39,628 WARN  [GossipStage:1] CassandraDiscovery.java:665 onRemove Removing node ip=/10.4.3.7 node={10.4.3.7}{662fa76b-01a9-4a37-ac46-c72a9e3e0811}{662fa76b-01a9-4a37-ac46-c72a9e3e081$
2020-12-14 07:02:39,722 INFO  [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] MasterService.java:245 runTasks node-removed-/10.4.3.7[org.elassandra.discovery.CassandraDiscovery$RoutingTableU$
2020-12-14 07:02:39,723 INFO  [elasticsearch[10.4.3.8][clusterApplierService#updateTask][T#1]] ClusterApplierService.java:487 applyChanges removed {{10.4.3.7}{662fa76b-01a9-4a37-ac46-c72a9e3e0811}{66$
2020-12-17 07:02:39,846 ERROR [GossipTasks:1] NoSpamLogger.java:97 log java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
        at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178)
        at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465)
        at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895)
        at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78)
        at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240)
        at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
        at java.lang.Thread.run(Thread.java:748)

After that im not longer able to delete existing elastic indexes:

2020-12-23 16:13:49,917 WARN  [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] CassandraDiscovery.java:1125 publishAsParticipator Ignore and acknowlegde obsolete update metadata=662fa76b-01a9-4a37-ac46-c72a9e3e0811/137
2020-12-23 16:13:49,931 WARN  [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] AppliedClusterStateAction.java:95 sendAppliedToNode error sending cluster state commit state [662fa76b-01a9-4a37-ac46-c72a9e3e0811/137] to null
java.lang.NullPointerException: discovery node must not be null
        at java.util.Objects.requireNonNull(Objects.java:228)
        at org.elasticsearch.transport.TransportService.isLocalNode(TransportService.java:1223)
        at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:531)
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:510)
        at org.elassandra.discovery.AppliedClusterStateAction.sendAppliedToNode(AppliedClusterStateAction.java:78)
        at org.elassandra.discovery.CassandraDiscovery.publishAsParticipator(CassandraDiscovery.java:1126)
        at org.elassandra.discovery.CassandraDiscovery.publish(CassandraDiscovery.java:960)
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:251)
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:152)
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-12-23 16:14:04,682 WARN  [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] CassandraDiscovery.java:1125 publishAsParticipator Ignore and acknowlegde obsolete update metadata=662fa76b-01a9-4a37-ac46-c72a9e3e0811/137
2020-12-23 16:14:04,683 WARN  [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] AppliedClusterStateAction.java:95 sendAppliedToNode error sending cluster state commit state [662fa76b-01a9-4a37-ac46-c72a9e3e0811/137] to null
java.lang.NullPointerException: discovery node must not be null

How can I fix the indexes / elastic now?

vroyer commented 3 years ago

Update elassandra to the last version.