search

strapdata / elassandra

Elassandra = Elasticsearch + Apache Cassandra
http://www.elassandra.io
Apache License 2.0
1.71k stars 198 forks source link

the maximum number of indexes created for a cluster #372

Open tengzhuofei opened 3 years ago

tengzhuofei commented 3 years ago

elassandra 6.8.4.10

With the compression parameter turned on, the maximum number of indexes created for a cluster can only be about 5100?

vroyer commented 3 years ago

Could you please provide information to reproduce the same test and hit this limit of 5100. Of course, long index names can shorten this limit.

On 21 Sep 2020, at 04:49, tengzhuofei notifications@github.com wrote:

elassandra 6.8.4.10

With the compression parameter turned on, the maximum number of indexes created for a cluster can only be about 5100?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/strapdata/elassandra/issues/372, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOMPGJE46MNSOSDNAPQQSLSG25MPANCNFSM4RUAQTIA.

tengzhuofei commented 3 years ago

Hi! I just kept creating indexes, and did not store data.

vroyer commented 3 years ago

Yes, but if you created 5000 underlying Cassandra keyspaces, that’s a Cassandra limitation….

On 21 Sep 2020, at 11:15, tengzhuofei notifications@github.com wrote:

Hi! I just kept creating indexes, and did not store data. — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/strapdata/elassandra/issues/372#issuecomment-695996999, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOMPGJFFUCMCPE5SFSS36DSG4KSTANCNFSM4RUAQTIA.

tengzhuofei commented 3 years ago

I only created 3 keyspaces and set the keyspace value through the setting of elasticsearch. There is no limit to the content of Gossip X1. I have enabled compression?

tengzhuofei commented 3 years ago

I restart a pod and too many pending states error

2020-09-21 11:16:20,135 DEBUG [elasticsearch[10.8.2.86][masterService#updateTask][T#1]] org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:255) publishing cluster state version [1259] metadata=[29791b59-4b87-41ff-84e1-3c73c67ecf21/5114] 2020-09-21 11:16:20,135 DEBUG [elasticsearch[10.8.2.86][masterService#updateTask][T#1]] org.elassandra.discovery.CassandraDiscovery.publishLocalUpdate(CassandraDiscovery.java:1196) Local update source=shard-started-update-routing index=twin_943[org.elassandra.shard.CassandraShardStateListener$RoutingTableUpdateTask@2764ffc9] metadata=29791b59-4b87-41ff-84e1-3c73c67ecf21/5114 2020-09-21 11:16:20,135 WARN [elasticsearch[10.8.2.86][masterService#updateTask][T#1]] org.elassandra.discovery.PendingClusterStatesQueue.addPending(PendingClusterStatesQueue.java:90) dropping pending state.version=[235] more than [1024] pending states. 2020-09-21 11:16:20,135 DEBUG [elasticsearch[10.8.2.86][generic][T#5]] org.elasticsearch.index.engine.CombinedDeletionPolicy.updateRetentionPolicy(CombinedDeletionPolicy.java:99) Safe commit [CommitPoint{segment[segments_2], userData[{history_uuid=Oi3NwB7ISseTUo7C-2ZMfw, local_checkpoint=-1, max_seq_no=-1, max_unsafe_auto_id_timestamp=-1, translog_generation=1, translog_uuid=18R8dd0eRRm0s-hYRDZVDA}]}], last commit [CommitPoint{segment[segments_2], userData[{history_uuid=Oi3NwB7ISseTUo7C-2ZMfw, local_checkpoint=-1, max_seq_no=-1, max_unsafe_auto_id_timestamp=-1, translog_generation=1, translog_uuid=18R8dd0eRRm0s-hYRDZVDA}]}] 2020-09-21 11:16:20,135 WARN [elasticsearch[10.8.2.86][masterService#updateTask][T#1]] org.elassandra.discovery.CassandraDiscovery$4.onNewClusterStateFailed(CassandraDiscovery.java:1212) failed while applying cluster state locally source=shard-started-update-routing index=feature_134[org.elassandra.shard.CassandraShardStateListener$RoutingTableUpdateTask@5555feb2] org.elasticsearch.ElasticsearchException: too many pending states ([1024] pending) at org.elassandra.discovery.PendingClusterStatesQueue.addPending(PendingClusterStatesQueue.java:91) at org.elassandra.discovery.CassandraDiscovery.publishLocalUpdate(CassandraDiscovery.java:1198) at org.elassandra.discovery.CassandraDiscovery.publish(CassandraDiscovery.java:996) at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:257) at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:158) at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)