mcasas993
commented
2 weeks ago The test was about one hour with the command manager running with ./gradlew run --debug and testing the endpoint. The test can't reproduce the mentioned error. Probably the error was in a previous version of the plugin.
Description
Epic: https://github.com/wazuh/wazuh-indexer/issues/349
When the Command Manager plugin runs for some time, this error starts to appear in the console and the cluster becomes unstable / unresponsive.
Details
```log 2024-10-16T12:52:25.207+0200 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 2024-10-16T12:52:25.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:52:25.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:52:25.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:52:25.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:52:25.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:52:25.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:52:22.876+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 2024-10-16T12:52:22.876+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:52:27,935][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:52:27.942+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:52:32,936][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:52:32.949+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:52:33.996+0200 [LIFECYCLE] [org.gradle.internal.resources.AbstractTrackedResourceLock] 2024-10-16T12:52:33.996+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker Thread 3: acquired lock on worker lease 2024-10-16T12:52:33.996+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker Thread 3: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 3: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 3: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 8: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 8: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 7: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 7: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 2: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 2: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 9: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 9: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 6: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 6: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 5: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 5: released lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 4: acquired lock on worker lease 2024-10-16T12:52:33.997+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 4: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 12: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 12: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 10: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 10: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 11: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 11: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 15: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 15: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 14: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 14: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 13: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Execution worker Thread 13: released lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] included builds: acquired lock on worker lease 2024-10-16T12:52:33.998+0200 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] included builds: released lock on worker lease 2024-10-16T12:52:35.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:52:35.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:52:35.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:52:35.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:52:35.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:52:35.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:52:32.877+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 2024-10-16T12:52:32.877+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run 2024-10-16T12:52:37.954+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:52:37,937][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.980+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:52:37.981+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:52:42.987+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:52:42,981][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.988+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.989+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.989+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.989+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:52:42.989+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:52:42.989+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:52:42.989+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:52:45.207+0200 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 2024-10-16T12:52:45.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:52:45.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:52:45.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:52:45.208+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:52:45.209+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:52:45.209+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. ```
When this happens, executing a
POST /_plugins/_command_manager/commandsrequest returns the following message:Full log
```log 2024-10-16T12:57:05.076+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 2024-10-16T12:57:05.076+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:08,048][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:57:08.065+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:57:09.166+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:09,110][INFO ][c.w.c.r.a.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [4] from host [127.0.0.1:9200] 2024-10-16T12:57:09.166+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:09,111][INFO ][c.w.c.u.h.HttpRestClient ] [integTest-0] Sending payload with id [KG75lJIBRAPV3wyYcw33] to [https://httpbin.org/post] 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:09,646][INFO ][c.w.c.r.a.RestPostCommandAction] [integTest-0] Received response to POST request with code [200] 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:09,646][INFO ][c.w.c.r.a.RestPostCommandAction] [integTest-0] Raw response: 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "args": {}, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "data": "{\"agent\":{\"groups\":[\"groups000\"]},\"command\":{\"source\":\"Engine\",\"user\":\"user53\",\"target\":{\"type\":\"agent\",\"id\":\"target4\"},\"action\":{\"name\":\"restart\",\"args\":[\"/path/to/executable/arg6\",\"/path/to/executable/arg6\"],\"version\":\"v4\"},\"timeout\":30,\"status\":\"PENDING\",\"order_id\":\"J275lJIBRAPV3wyYcw33\",\"request_id\":\"Jm75lJIBRAPV3wyYcw33\"}}", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "files": {}, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "form": {}, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "headers": { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "Content-Type": "application/json; charset=UTF-8", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "Host": "httpbin.org", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "Transfer-Encoding": "chunked", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "User-Agent": "Apache-HttpAsyncClient/5.4 (Java/21.0.4)", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "X-Amzn-Trace-Id": "Root=1-670f9c05-7273cc85159c97394b1259c4" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] }, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "json": { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "agent": { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "groups": [ 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "groups000" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] ] 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] }, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "command": { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "action": { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "args": [ 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "/path/to/executable/arg6", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "/path/to/executable/arg6" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] ], 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "name": "restart", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "version": "v4" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] }, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "order_id": "J275lJIBRAPV3wyYcw33", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "request_id": "Jm75lJIBRAPV3wyYcw33", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "source": "Engine", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "status": "PENDING", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "target": { 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "id": "target4", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "type": "agent" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] }, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "timeout": 30, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "user": "user53" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] } 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] }, 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "origin": "79.117.251.93", 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] "url": "https://httpbin.org/post" 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] } 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:09,647][INFO ][c.w.c.i.CommandIndex ] [integTest-0] Index template index-template-commands already exists. Skipping creation. 2024-10-16T12:57:09.667+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:09,647][INFO ][c.w.c.i.CommandIndex ] [integTest-0] Indexing command with id [KG75lJIBRAPV3wyYcw33] 2024-10-16T12:57:10.268+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:10,182][DEBUG][o.o.c.c.ElectionSchedulerFactory] [integTest-0] scheduleNextElection{gracePeriod=500ms, thisAttempt=74, maxDelayMillis=7500, delayMillis=5969, ElectionScheduler{attempt=75, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election 2024-10-16T12:57:10.268+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:10,182][DEBUG][o.o.c.c.ElectionSchedulerFactory] [integTest-0] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=75, maxDelayMillis=7600, delayMillis=8069, ElectionScheduler{attempt=76, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} 2024-10-16T12:57:10.268+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:10,182][DEBUG][o.o.c.c.Coordinator ] [integTest-0] skip prevoting as local node is unhealthy: [health check failed due to broken node lock] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:10,745][ERROR][o.o.m.f.FsHealthService ] [integTest-0] health check failed 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:900) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:212) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:195) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:57:10.769+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:57:11.470+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:11,445][WARN ][o.o.c.c.ClusterFormationFailureHelper] [integTest-0] this node is unhealthy: health check failed due to broken node lock 2024-10-16T12:57:13.072+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:13,050][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:57:13.073+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:57:15.207+0200 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager] 2024-10-16T12:57:15.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:57:15.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:57:15.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:57:15.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry. 2024-10-16T12:57:15.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry. 2024-10-16T12:57:15.207+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry. 2024-10-16T12:57:12.976+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] 2024-10-16T12:57:12.976+0200 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :run 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:18,051][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.080+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:57:18.081+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-10-16T12:57:18.281+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:18,252][DEBUG][o.o.c.c.ElectionSchedulerFactory] [integTest-0] scheduleNextElection{gracePeriod=500ms, thisAttempt=75, maxDelayMillis=7600, delayMillis=8069, ElectionScheduler{attempt=76, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} starting election 2024-10-16T12:57:18.281+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:18,252][DEBUG][o.o.c.c.ElectionSchedulerFactory] [integTest-0] scheduling scheduleNextElection{gracePeriod=500ms, thisAttempt=76, maxDelayMillis=7700, delayMillis=6231, ElectionScheduler{attempt=77, ElectionSchedulerFactory{initialTimeout=100ms, backoffTime=100ms, maxTimeout=10s}}} 2024-10-16T12:57:18.281+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:18,253][DEBUG][o.o.c.c.Coordinator ] [integTest-0] skip prevoting as local node is unhealthy: [health check failed due to broken node lock] 2024-10-16T12:57:21.485+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:21,446][WARN ][o.o.c.c.ClusterFormationFailureHelper] [integTest-0] this node is unhealthy: health check failed due to broken node lock 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] [2024-10-16T12:57:23,053][WARN ][o.o.t.ThreadPool ] [integTest-0] failed to run scheduled task [org.opensearch.node.resource.tracker.AverageIoUsageTracker$$Lambda/0x0000786dc8861920@75e850d] on thread pool [generic] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2024-10-16T09:46:40.106129231Z, (lock=NativeFSLock(path=/home/alex/wazuh/wazuh-indexer-plugins/plugins/command-manager/build/testclusters/integTest-0/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-10-16T09:39:08.20723354Z)) 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1149) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.env.NodeEnvironment.nodePaths(NodeEnvironment.java:930) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsProbe.stats(FsProbe.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:93) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:113) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService$FsInfoCache.refresh(FsService.java:100) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.monitor.fs.FsService.stats(FsService.java:88) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.preValidateFsStats(AverageIoUsageTracker.java:87) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.getUsage(AverageIoUsageTracker.java:50) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.node.resource.tracker.AverageIoUsageTracker.lambda$doStart$0(AverageIoUsageTracker.java:77) ~[opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:246) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:941) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.16.0.jar:2.16.0] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] 2024-10-16T12:57:23.087+0200 [LIFECYCLE] [org.opensearch.gradle.testclusters.RunTask] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] ```
Plugin is running in debug mode:
gradlew run --debug