search

opensearch-project / cross-cluster-replication

Synchronize your data across multiple clusters for lower latencies and higher availability
https://opensearch.org/docs/latest/replication-plugin/index/
Apache License 2.0
48 stars 58 forks source link

[BUG] remotes set in opensearch.yml do not work #1431

Open uiteindelijkwordtallesbagger opened 3 weeks ago

uiteindelijkwordtallesbagger commented 3 weeks ago

What is the bug? If I setup a remote via opensearch.yml. The replication-plugin will not work with it. Please either fix this or make it less obscure in the error log/documentation.

How can one reproduce the bug? Steps to reproduce the behavior: Follow the documentation on https://opensearch.org/docs/latest/tuning-your-cluster/replication-plugin/index/ Except for the step: Set up a cross-cluster connection

Instead edit opensearch.yml 

cluster.remote.my-connection-alias: 
       mode: "proxy"
       proxy_address: "172.22.0.3:9300"

seed gives the same error.

check GET /_remote/info And see that it looks the same as in the documentation. Follow the documentation. Replication wont start(timeout)

What is the expected behavior? Because _remote/info looks alike I expect the system to be in the same state as it would have if I used the api.

What is your host/environment?

Do you have any screenshots? no

Do you have any additional context?

[2024-08-26T16:44:44,015][INFO ][o.o.r.a.i.TransportReplicateIndexAction] [serverA] Setting-up replication for serverB:repl-test-leader-01 -> repl-test-follower-01
[2024-08-26T16:44:44,016][INFO ][o.o.r.a.s.TransportValidatePermissionsAction] [serverA] Replication setup - Permissions validation successful for Index - repl-test-follower-01 and role all_access
[2024-08-26T16:44:44,017][INFO ][o.o.r.a.s.TransportSetupChecksAction] [serverA] Permissions validation successful for User [local:serverA, resource:repl-test-follower-01]
[2024-08-26T16:44:44,023][INFO ][o.o.r.a.s.TransportSetupChecksAction] [serverA] Permissions validation successful for User [connection:serverB, resource:repl-test-leader-01]
[2024-08-26T16:44:44,024][INFO ][o.o.r.a.s.TransportValidatePermissionsAction] [serverA] Replication setup - Permissions validation successful for Index - repl-test-follower-01 and role all_access
[2024-08-26T16:44:44,024][INFO ][o.o.r.a.s.TransportSetupChecksAction] [serverA] Permissions validation successful for User [connection:serverB, resource:repl-test-leader-01]
[2024-08-26T16:44:44,027][INFO ][o.o.r.a.s.TransportSetupChecksAction] [serverA] Permissions validation successful for role [connection:serverB, resource:repl-test-leader-01]
[2024-08-26T16:44:44,055][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [serverA] Detected cluster change event for destination migration
[2024-08-26T16:44:44,056][WARN ][o.o.c.s.ClusterApplierService] [serverA] failed to notify ClusterStateListener
java.lang.NullPointerException: p must not be null
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.persistent.PersistentTasksService.lambda$waitForPersistentTaskCondition$1(PersistentTasksService.java:181) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:225) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-08-26T16:44:44,057][WARN ][o.o.c.s.ClusterApplierService] [serverA] failed to notify ClusterStateListener
java.lang.NullPointerException: p must not be null
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.persistent.PersistentTasksService.lambda$waitForPersistentTaskCondition$1(PersistentTasksService.java:181) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:225) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-08-26T16:44:44,065][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [serverA] Detected cluster change event for destination migration
[2024-08-26T16:44:44,073][ERROR][o.o.r.t.i.IndexReplicationTask] [serverA] [repl-test-follower-01] Unable to initiate restore call for repl-test-follower-01 from serverB:repl-test-leader-01
org.opensearch.snapshots.SnapshotRestoreException: [replication-remote-repo-serverB:replication-remote-snapshot] snapshot does not exist
        at org.opensearch.snapshots.RestoreService.lambda$restoreSnapshot$2(RestoreService.java:242) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112) ~[opensearch-2.10.0.jar:2.10.0]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:160) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:141) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.action.StepListener.innerOnResponse(StepListener.java:79) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.core.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:55) ~[opensearch-core-2.10.0.jar:2.10.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:2037) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-08-26T16:44:44,073][ERROR][o.o.r.a.i.TransportReplicateIndexClusterManagerNodeAction] [serverA] Failed to trigger replication for repl-test-follower-01 - java.lang.IllegalStateException: Timed out when waiting for persistent task after 30s
        at org.opensearch.persistent.PersistentTasksService$WaitForPersistentTaskListener.onTimeout(PersistentTasksService.java:255)
        at org.opensearch.persistent.PersistentTasksService$1.onTimeout(PersistentTasksService.java:203)
        at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)
        at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

[2024-08-26T16:44:44,074][WARN ][r.suppressed             ] [serverA] path: /_plugins/_replication/repl-test-follower-01/_start, params: {pretty=true, index=repl-test-follower-01}
java.lang.IllegalStateException: Timed out when waiting for persistent task after 30s
        at org.opensearch.persistent.PersistentTasksService$WaitForPersistentTaskListener.onTimeout(PersistentTasksService.java:255) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.persistent.PersistentTasksService$1.onTimeout(PersistentTasksService.java:203) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2024-08-26T16:44:44,086][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [serverA] Detected cluster change event for destination migration
[2024-08-26T16:44:44,086][ERROR][o.o.r.t.i.IndexReplicationTask] [serverA] [repl-test-follower-01] Moving replication[IndexReplicationTask:7696][reason=Unable to initiate restore call for repl-test-follower-01 from serverB:repl-test-leader-01] to failed state
[2024-08-26T16:44:44,099][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [serverA] Detected cluster change event for destination migration
[2024-08-26T16:44:44,100][WARN ][o.o.p.AllocatedPersistentTask] [serverA] task replication:index:repl-test-follower-01 failed with an exception
java.lang.RuntimeException: Unable to initiate restore call for repl-test-follower-01 from serverB:repl-test-leader-01
        at org.opensearch.replication.task.index.IndexReplicationTask$failReplication$2.invokeSuspend(IndexReplicationTask.kt:281) [opensearch-cross-cluster-replication-2.10.0.0.jar:2.10.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-08-26T16:44:44,101][INFO ][o.o.r.t.i.IndexReplicationTask] [serverA] [repl-test-follower-01] Going to mark IndexReplicationTask:7696 task as completed
[2024-08-26T16:44:44,101][WARN ][o.o.p.AllocatedPersistentTask] [serverA] attempt to complete task [cluster:indices/admin/replication[c]] with id [replication:index:repl-test-follower-01] in the [COMPLETED] state
[2024-08-26T16:44:44,101][ERROR][o.o.r.t.i.IndexReplicationTask] [serverA] [repl-test-follower-01] Task failed due to java.lang.RuntimeException: Unable to initiate restore call for repl-test-follower-01 from serverB:repl-test-leader-01
        at org.opensearch.replication.task.index.IndexReplicationTask$failReplication$2.invokeSuspend(IndexReplicationTask.kt:281)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147)
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

[2024-08-26T16:44:44,101][INFO ][o.o.r.t.i.IndexReplicationTask] [serverA] [repl-test-follower-01] Verifying task details - currentTask={isAssigned=true,executorNode=InYeAouPQ0aq8a7LgEQ24A}
[2024-08-26T16:44:44,101][WARN ][o.o.p.PersistentTasksClusterService] [serverA] persistent task replication:index:repl-test-follower-01 failed
java.lang.RuntimeException: Unable to initiate restore call for repl-test-follower-01 from serverB:repl-test-leader-01
        at org.opensearch.replication.task.index.IndexReplicationTask$failReplication$2.invokeSuspend(IndexReplicationTask.kt:281) ~[?:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[?:?]
        at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147) ~[?:?]
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) ~[?:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) ~[?:?]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) ~[?:?]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-08-26T16:44:44,102][INFO ][o.o.r.t.i.IndexReplicationTask] [serverA] [repl-test-follower-01] Successfully persisted task status
[2024-08-26T16:44:44,108][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [serverA] Detected cluster change event for destination migration
[2024-08-26T16:44:44,108][WARN ][o.o.c.s.ClusterApplierService] [serverA] failed to notify ClusterStateListener
java.lang.NullPointerException: p must not be null
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.persistent.PersistentTasksService.lambda$waitForPersistentTaskCondition$1(PersistentTasksService.java:181) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:225) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-08-26T16:44:44,109][WARN ][o.o.c.s.ClusterApplierService] [serverA] failed to notify ClusterStateListener
java.lang.NullPointerException: p must not be null
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.replication.util.CoroutinesKt$waitForTaskCondition$2$1.test(Coroutines.kt:163) ~[?:?]
        at org.opensearch.persistent.PersistentTasksService.lambda$waitForPersistentTaskCondition$1(PersistentTasksService.java:181) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:225) ~[opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:625) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:613) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:577) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.10.0.jar:2.10.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.10.0.jar:2.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
dblock commented 1 day ago

[Catch All Triage - 1, 2, 3, 4, 5]