search

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
152 stars 113 forks source link

Flaky test KNNCreateIndexFromModelTests > testCreateIndexFromModel #1522

Open jmazanec15 opened 6 months ago

jmazanec15 commented 6 months ago

Flaky test coming from: https://github.com/opensearch-project/k-NN/actions/runs/8192371748/job/22403731973?pr=1520.

From error, I see:

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.KNNCreateIndexFromModelTests.testCreateIndexFromModel" -Dtests.seed=AF6F47FAA64F089E -Dtests.security.manager=false -Dtests.locale=nn-NO -Dtests.timezone=Australia/West -Druntime.java=21
> Task :test

org.opensearch.knn.index.KNNCreateIndexFromModelTests > testCreateIndexFromModel FAILED
Suite: Test class org.opensearch.knn.index.KNNCreateIndexFromModelTests
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=42, name=opensearch[node_s_0][generic][T#3], state=RUNNABLE, group=TGRP-KNNCreateIndexFromModelTests]
  2> mar 08, 2024 1:56:14 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
        at __randomizedtesting.SeedInfo.seed([AF6F47FAA64F089E:8B5CE437AA9B6271]:0)
  2> INFO: Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false

  2> mar 08, 2024 1:56:20 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
        Caused by:
        java.lang.RuntimeException: java.util.concurrent.ExecutionException: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
  2> WARNING: Uncaught exception in thread: Thread[#42,opensearch[node_s_0][generic][T#3],5,TGRP-KNNCreateIndexFromModelTests]
            at __randomizedtesting.SeedInfo.seed([AF6F47FAA64F089E]:0)
            at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:92)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.base/java.lang.Thread.run(Thread.java:1583)

            Caused by:
            java.util.concurrent.ExecutionException: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
                at org.opensearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:286)
                at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:273)
                at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:104)
                at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.lambda$get$9(ModelDao.java:384)
                at org.opensearch.knn.indices.ModelDao.runWithStashedThreadContext(ModelDao.java:705)
                at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.get(ModelDao.java:379)
                at org.opensearch.knn.training.TrainingJobClusterStateListener.updateModelsNewCluster(TrainingJobClusterStateListener.java:112)
  2> java.lang.RuntimeException: java.util.concurrent.ExecutionException: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
                at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:90)
  2>    at __randomizedtesting.SeedInfo.seed([AF6F47FAA64F089E]:0)
                ... 4 more
  2>    at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:92)

  2>    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
                Caused by:
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
                [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  2>    at java.base/java.lang.Thread.run(Thread.java:1583)
  2> Caused by: java.util.concurrent.ExecutionException: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
  2>    at org.opensearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:286)
  2>    at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:273)
  2>    at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:104)
  2>    at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.lambda$get$9(ModelDao.java:384)
  2>    at org.opensearch.knn.indices.ModelDao.runWithStashedThreadContext(ModelDao.java:705)
  2>    at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.get(ModelDao.java:379)
  2>    at org.opensearch.knn.training.TrainingJobClusterStateListener.updateModelsNewCluster(TrainingJobClusterStateListener.java:112)
  2>    at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:90)
  2>    ... 4 more
  2> Caused by: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
  2>    at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:284)
  2>    at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:202)
  2>    at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:197)
  2>    at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:423)
  2>    at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:185)
  2>    at org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:125)
  2>    at org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:78)
  2>    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
  2>    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
  2>    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
  2>    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
  2>    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
  2>    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:482)
  2>    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:469)
  2>    at org.opensearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:66)
  2>    ... 9 more

  2> REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.KNNCreateIndexFromModelTests.testCreateIndexFromModel" -Dtests.seed=AF6F47FAA64F089E -Dtests.security.manager=false -Dtests.locale=nn-NO -Dtests.timezone=Australia/West -Druntime.java=21
  2> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=42, name=opensearch[node_s_0][generic][T#3], state=RUNNABLE, group=TGRP-KNNCreateIndexFromModelTests]
        at __randomizedtesting.SeedInfo.seed([AF6F47FAA64F089E:8B5CE437AA9B6271]:0)

        Caused by:
        java.lang.RuntimeException: java.util.concurrent.ExecutionException: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
            at __randomizedtesting.SeedInfo.seed([AF6F47FAA64F089E]:0)
            at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:92)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.base/java.lang.Thread.run(Thread.java:1583)

            Caused by:
            java.util.concurrent.ExecutionException: [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
                at org.opensearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:286)
                at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:273)
                at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:104)
                at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.lambda$get$9(ModelDao.java:384)
                at org.opensearch.knn.indices.ModelDao.runWithStashedThreadContext(ModelDao.java:705)
                at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.get(ModelDao.java:379)
                at org.opensearch.knn.training.TrainingJobClusterStateListener.updateModelsNewCluster(TrainingJobClusterStateListener.java:112)
                at org.opensearch.knn.training.TrainingJobClusterStateListener.lambda$clusterChanged$0(TrainingJobClusterStateListener.java:90)
                ... 4 more

                Caused by:
                [.opensearch-knn-models] IndexNotFoundException[no such index [.opensearch-knn-models]]
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:284)
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:202)
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:197)
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:423)
                    at app//org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:185)
                    at app//org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:125)
                    at app//org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:78)
                    at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
                    at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
                    at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
                    at app//org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
                    at app//org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
                    at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:482)
                    at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:469)
                    at app//org.opensearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:66)
                    ... 9 more
  2> NOTE: test params are: codec=Asserting(Lucene99): {index_uuid=Lucene99, type=PostingsFormat(name=Asserting)}, docValues:{}, maxPointsInLeafNode=822, maxMBSortInHeap=6.4839395353027305, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=nn-NO, timezone=Australia/West
  2> NOTE: Mac OS X 12.7.3 x86_64/Azul Systems, Inc. 21.0.2 (64-bit)/cpus=4,threads=1,free=310093232,total=536870912
  2> NOTE: All tests run in this JVM: [KNNInvalidIndicesExceptionTests, IndexUtilTests, KNNCreateIndexFromModelTests]
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:284)
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:202)
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:197)
                    at app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:423)
                    at app//org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:185)
                    at app//org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:125)
                    at app//org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:78)
                    at app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
                    at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
                    at app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
                    at app//org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
                    at app//org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
                    at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:482)
                    at app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:469)
                    at app//org.opensearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:66)
                    ... 9 more
jmazanec15 commented 6 months ago

Another instance https://github.com/opensearch-project/k-NN/actions/runs/8267747650/job/22618983008?pr=1533