search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
8.89k stars 1.63k forks source link

[BUG] org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdatedEvenForMisconfiguredReplicas if flaky #13737

Open reta opened 2 weeks ago

reta commented 2 weeks ago

Describe the bug

The test case org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdatedEvenForMisconfiguredReplicas is flaky:

java.lang.AssertionError: shard [migration-index-allocation-exclude][0] is not locked
    at __randomizedtesting.SeedInfo.seed([32BC8AFD872402D]:0)
    at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
    at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1247)
    at org.opensearch.index.IndexService.onShardClose(IndexService.java:719)
    at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:842)
    at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:829)
    at org.opensearch.index.store.Store.closeInternal(Store.java:573)
    at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
    at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
    at org.opensearch.index.store.Store.decRef(Store.java:546)
    at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1868)
    at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1844)
    at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4648)
    at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1067)
    at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1211)
    at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Standard Output

五月 17, 2024 8:55:54 下午 com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
警告: Uncaught exception in thread: Thread[#4943,opensearch[node_t4][refresh][T#1],5,TGRP-RemoteMigrationIndexMetadataUpdateIT]
java.lang.AssertionError: shard [migration-index-allocation-exclude][0] is not locked
    at __randomizedtesting.SeedInfo.seed([32BC8AFD872402D]:0)
    at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
    at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1247)
    at org.opensearch.index.IndexService.onShardClose(IndexService.java:719)
    at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:842)
    at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:829)
    at org.opensearch.index.store.Store.closeInternal(Store.java:573)
    at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
    at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
    at org.opensearch.index.store.Store.decRef(Store.java:546)
    at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1868)
    at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1844)
    at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4648)
    at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1067)
    at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1211)
    at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583

Related component

Storage:Remote

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotemigration.RemoteMigrationIndexMetadataUpdateIT.testIndexSettingsUpdatedEvenForMisconfiguredReplicas" -Dtests.seed=32BC8AFD872402D

Expected behavior

The test must always pass

Additional Details

Plugins Standard

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

Additional context

reta commented 2 weeks ago

Introduced by https://github.com/opensearch-project/OpenSearch/pull/13316, @shourya035 please prioritize

sachinpkale commented 2 days ago

[Storage Triage - attendees 1 2 3 4 5 6 7 8 9 10 ]

@shourya035 Please check if you can get this done by 2.15