search

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data
https://opensearch.org/docs/latest/search-plugins/knn/index/
Apache License 2.0
155 stars 114 forks source link

Flaky test KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed on machines with more than 4 CPUs on 2.11.1 #1392

Closed salyh closed 8 months ago

salyh commented 9 months ago

What is the bug?

Unit test KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed fails on machines with more than 4 CPUs. On machines with fewer cores its green and looks stable

 org.opensearch.knn.index.KNNSettingsTests > testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed FAILED
    java.lang.AssertionError: no warnings, expected: [[index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version., no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release]
        at __randomizedtesting.SeedInfo.seed([F01086BC8EEF63CB:61987A4E1BA5D039]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertNotNull(Assert.java:713)
        at org.opensearch.test.OpenSearchTestCase.assertWarnings(OpenSearchTestCase.java:578)
        at org.opensearch.test.OpenSearchTestCase.assertWarnings(OpenSearchTestCase.java:564)
        at org.opensearch.test.OpenSearchTestCase.assertWarnings(OpenSearchTestCase.java:518)
        at org.opensearch.knn.index.KNNSettingsTests.assertWarnings(KNNSettingsTests.java:172)
        at
org.opensearch.knn.index.KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed(KNNSettingsTests.java:90)
  1> [2024-01-18T09:56:09,889][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed] closing ...
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertNotNull(Assert.java:713)
        at org.opensearch.test.OpenSearchTestCase.assertWarnings(OpenSearchTestCase.java:578)
        at org.opensearch.test.OpenSearchTestCase.assertWarnings(OpenSearchTestCase.java:564)
        at org.opensearch.test.OpenSearchTestCase.assertWarnings(OpenSearchTestCase.java:518)
        at org.opensearch.knn.index.KNNSettingsTests.assertWarnings(KNNSettingsTests.java:172)
        at org.opensearch.knn.index.KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed(KNNSettingsTests.java:90)

 Tests with failures:
 - org.opensearch.knn.index.KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed
461 tests completed, 1 failed, 2 skipped

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Run the unit tests on a machine with 8 cpu cores

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed" -Dtests.seed=F01086BC8EEF63CB -Dtests.security.manager=false -Dtests.locale=ar-KW -Dtests.timezone=Australia/Darwin -Druntime.java=17

What is the expected behavior? Test succeeds

What is your host/environment?

Do you have any additional context? There are a few other tests flakythe same way. I open extra issues for them.

junqiu-lei commented 8 months ago

Hi @salyh, I tried to reproduce the issue on 8 CPUs desktop on Amazon Linux 2 OS, however all tests passed in KNNSettingsTests, no failure found.

Tag with other related issues: https://github.com/opensearch-project/k-NN/issues/1391, https://github.com/opensearch-project/k-NN/issues/1390, https://github.com/opensearch-project/k-NN/issues/1389

âžś  k-NN git:(2.11) ./gradlew ':test' --tests "org.opensearch.knn.index.KNNSettingsTests.*" -Dtests.seed=F01086BC8EEF63CB -Dtests.security.manager=false -Dtests.locale=ar-KW -Dtests.timezone=Australia/Darwin -Druntime.java=17
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6.1
  OS Info               : Linux 5.10.205-172.804.amzn2int.x86_64 (amd64)
  Runtime JDK Version   : 17 (Amazon Corretto JDK)
  Runtime java.home     : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Gradle JDK Version    : 17 (Amazon Corretto JDK)
  Gradle java.home      : /usr/lib/jvm/java-17-amazon-corretto.x86_64
  Random Testing Seed   : F01086BC8EEF63CB
  In FIPS 140 mode      : false
=======================================
...

> Task :buildJniLib
[ 89%] Built target NonMetricSpaceLib
[ 94%] Built target opensearchknn_common
[100%] Built target opensearchknn_nmslib
[ 93%] Built target faiss
[ 96%] Built target opensearchknn_common
[100%] Built target opensearchknn_faiss

> Task :test
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 24s
15 actionable tasks: 6 executed, 9 up-to-date

âžś  k-NN git:(2.11) git log
commit ba09ac278595d1c3ca6e55c6558ab0171a392be0 (HEAD -> 2.11, origin/2.11)
Author: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Date:   Mon Nov 20 14:44:06 2023 -0800
salyh commented 8 months ago

Can you try running it a few times in a row?

junqiu-lei commented 8 months ago

@salyh Yes, I tried rerun for 10 times by both single unit test in KNNSettingsTests and KNNSettingsTests.*, all tests passed.

jmazanec15 commented 8 months ago

@salyh Can you try switching assertWarnings -> allowWarnings in the KNNSettingsTest file and see if it is resolved? I cant reproduce locally either.

For ref, here is example of assertWarnings: https://github.com/opensearch-project/k-NN/blob/main/src/test/java/org/opensearch/knn/index/KNNSettingsTests.java#L75

junqiu-lei commented 8 months ago

Hi @salyh, when you say it fails on machines with more than 4 CPUs, can I know if other environments(like OS) kept same in your tests in 4 CPUs machine and 8 CPUs machine? I'm trying to understand which situation exactly caused the issue, since @jmazanec15 and me weren’t able to reproduce it in our 8 CPUs machines.

jmazanec15 commented 8 months ago

Also @salyh can you paste output on failure from build/test-results/test/binary/output.bin file?

salyh commented 8 months ago

Some updates (on current k-NN main branch):

On Ubuntu 22.04 on t2.2xlarge EC2 : docker run -it opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 Then clone k-NN, init and update the submodules and run ./gradlew test

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.KNNSettingsTests.testGetEfSearch_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed" -Dtests.seed=C2A7F8EBCA643150 -Dtests.security.manager=false -Dtests.locale=it -Dtests.timezone=Atlantic/Bermuda -Druntime.java=21

  2> NOTE: Linux 6.2.0-1017-aws amd64/Eclipse Adoptium 21.0.1 (64-bit)/cpus=8,threads=1,free=397401296,total=536870912
  2> NOTE: All tests run in this JVM: [KNNInvalidIndicesExceptionTests, KNNCreateIndexFromModelTests, KNNSettingsTests]

build/test-results/test/binary/output.bin output.tar.gz

@junqiu-lei The 4 and 8 core machines are more or less identical

@jmazanec15 assertWarnings -> allowWarnings i try next

salyh commented 8 months ago

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.knn.index.KNNSettingsTests.testGetSettingValueDefault" -Dtests.seed=2AC3B8E3D4035575 -Dtests.security.manager=false -Dtests.locale=ar-MA -Dtests.timezone=Asia/Gaza -Druntime.java=21

output.tar.gz

salyh commented 8 months ago

@jmazanec15 assertWarnings -> allowWarnings leads to

Tests with failures:
 - org.opensearch.knn.index.KNNSettingsTests.testFilteredSearchAdvanceSetting_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed
 - org.opensearch.knn.index.KNNSettingsTests.testGetSettingValueDefault
 - org.opensearch.knn.index.KNNSettingsTests.testGetEfSearch_whenNoValuesProvidedByUsers_thenDefaultSettingsUsed
 - org.opensearch.knn.index.KNNSettingsTests.testGetEfSearch_whenEFSearchValueSetByUser_thenReturnValue
 - org.opensearch.knn.index.KNNSettingsTests.testGetSettingValueFromConfig
 - org.opensearch.knn.index.KNNSettingsTests.testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues

output.tar.gz

junqiu-lei commented 8 months ago

Thanks for providing more details on the reproduce steps. @salyh

However I still couldn’t reproduce the issue with multiple times run using the same env mentioned above:

OS: Ubuntu 22.04
EC2: t2.2xlarge
Docker Image: opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3

knn-settings-tests-output.log

cc: @jmazanec15

jmazanec15 commented 8 months ago

@salyh I was able to reproduce the issue. Below you can see the warning is in the one long but not other. Im not sure why this is happening. Given we arent really testing this, I think we should remove the assertWarnings altogether and then set enableWarningCheck in KNNTestCase to false and remove the assertWarning checks. @junqiu-lei can you update your PR to do this?

For the failure, I see

Suite: Test class org.opensearch.knn.index.KNNSettingsTests
  1> [2024-02-07T14:31:35,368][INFO ][o.o.k.i.KNNSettingsTests ] [testGetSettingValueDefault] before test
  1> [2024-02-07T14:31:35,428][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] version[2.11.1-SNAPSHOT], pid[2275], build[unknown/6b1986e964d440be9137eba1413015c31c5a7752/2023-11-29T21:17:26.630182988Z], OS[Linux/6.2.0-1017-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7]
  1> [2024-02-07T14:31:35,434][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] JVM home [/opt/java/openjdk-17]
  1> [2024-02-07T14:31:35,435][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] JVM arguments [-Dgradle.dist.lib=/home/ci-runner/.gradle/wrapper/dists/gradle-7.6.1-all/942lu1p9i6mhoyzmt401s4g74/gradle-7.6.1/lib, -Dgradle.user.home=/home/ci-runner/.gradle, -Dgradle.worker.jar=/home/ci-runner/.gradle/caches/7.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.library.path=/home/ci-runner/k-NN/jni/release, -Djava.locale.providers=SPI,COMPAT, -Djdk.attach.allowAttachSelf=true, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/home/ci-runner/k-NN/build/tmp/test/work, -Dorg.gradle.native=false, -Dtests.artifact=opensearch-knn, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=false, -Dtests.seed=1D74642C8A5A3ABE, -Dtests.task=:test, -XX:+HeapDumpOnOutOfMemoryError, -esa, --illegal-access=warn, -javaagent:../../tmp/expandedArchives/org.jacoco.agent-0.8.7.jar_3a83c50b4a016f281c4e9f3500d16b55/jacocoagent.jar=destfile=../../jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -XX:HeapDumpPath=/home/ci-runner/k-NN/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=US-ASCII, -Djava.io.tmpdir=/home/ci-runner/k-NN/build/testrun/test/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2024-02-07T14:31:35,436][WARN ][o.o.n.Node               ] [testGetSettingValueDefault] version [2.11.1-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
  1> [2024-02-07T14:31:35,437][INFO ][o.o.p.PluginsService     ] [testGetSettingValueDefault] no modules loaded
  1> [2024-02-07T14:31:35,438][INFO ][o.o.p.PluginsService     ] [testGetSettingValueDefault] loaded plugin [org.opensearch.knn.plugin.KNNPlugin]
  1> [2024-02-07T14:31:35,439][INFO ][o.o.p.PluginsService     ] [testGetSettingValueDefault] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
  1> [2024-02-07T14:31:35,439][INFO ][o.o.p.PluginsService     ] [testGetSettingValueDefault] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
  1> [2024-02-07T14:31:35,440][INFO ][o.o.e.ExtensionsManager  ] [testGetSettingValueDefault] ExtensionsManager initialized
  1> [2024-02-07T14:31:35,442][INFO ][o.o.e.NodeEnvironment    ] [testGetSettingValueDefault] using [1] data paths, mounts [[/ (overlay)]], net usable_space [475.7gb], net total_space [484.4gb], types [overlay]
  1> [2024-02-07T14:31:35,443][INFO ][o.o.e.NodeEnvironment    ] [testGetSettingValueDefault] heap size [512mb], compressed ordinary object pointers [true]
  1> [2024-02-07T14:31:35,445][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] node name [mock_ node], node ID [khb1hv__T_-1TgEdAAAAAA], cluster name [single-node-cluster-TEST_WORKER_VM=[4]-CLUSTER_SEED=[-7299691990438868214]-HASH=[11793E818C7]], roles [ingest, remote_cluster_client, data, cluster_manager]
  1> [2024-02-07T14:31:35,492][INFO ][o.o.d.DiscoveryModule    ] [testGetSettingValueDefault] using discovery type [zen] and seed hosts providers [settings]
  1> [2024-02-07T14:31:35,517][WARN ][o.o.g.DanglingIndicesState] [testGetSettingValueDefault] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2024-02-07T14:31:35,533][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] initialized
  1> [2024-02-07T14:31:35,533][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] starting ...
  1> [2024-02-07T14:31:35,543][INFO ][o.o.t.TransportService   ] [testGetSettingValueDefault] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
  1> [2024-02-07T14:31:35,544][INFO ][o.o.t.TransportService   ] [testGetSettingValueDefault] Remote clusters initialized successfully.
  1> [2024-02-07T14:31:35,583][INFO ][o.o.c.c.ClusterBootstrapService] [testGetSettingValueDefault] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing cluster-manager is discovered
  1> [2024-02-07T14:31:38,585][INFO ][o.o.c.c.Coordinator      ] [mock_ node] setting initial configuration to VotingConfiguration{khb1hv__T_-1TgEdAAAAAA}
  1> [2024-02-07T14:31:38,667][INFO ][o.o.c.s.MasterService    ] [mock_ node] elected-as-cluster-manager ([1] nodes joined)[{mock_ node}{khb1hv__T_-1TgEdAAAAAA}{M1AqfQa7Ss6ZWFmUO2xITA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{mock_ node}{khb1hv__T_-1TgEdAAAAAA}{M1AqfQa7Ss6ZWFmUO2xITA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
  1> [2024-02-07T14:31:38,679][INFO ][o.o.c.c.CoordinationState] [mock_ node] cluster UUID set to [qsOtR5etRb2NaDYVWFR69g]
  1> [2024-02-07T14:31:38,691][INFO ][o.o.c.s.ClusterApplierService] [mock_ node] cluster-manager node changed {previous [], current [{mock_ node}{khb1hv__T_-1TgEdAAAAAA}{M1AqfQa7Ss6ZWFmUO2xITA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
  1> [2024-02-07T14:31:38,693][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] started
  1> [2024-02-07T14:31:38,693][INFO ][o.o.d.PeerFinder         ] [mock_ node] setting findPeersInterval to [1s] as node commission status = [true] for local node [{mock_ node}{khb1hv__T_-1TgEdAAAAAA}{M1AqfQa7Ss6ZWFmUO2xITA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]
  1> [2024-02-07T14:31:38,693][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] stopping ...
  1> [2024-02-07T14:31:38,698][INFO ][o.o.g.GatewayService     ] [mock_ node] recovered [0] indices into cluster_state
  1> [2024-02-07T14:31:38,700][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] stopped
  1> [2024-02-07T14:31:38,700][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] closing ...
  1> [2024-02-07T14:31:38,702][INFO ][o.o.n.Node               ] [testGetSettingValueDefault] closed
  1> [2024-02-07T14:31:38,703][INFO ][o.o.k.i.KNNSettingsTests ] [testGetSettingValueDefault] after test

for a passing one, I see

1> [2024-02-07T14:31:38,732][INFO ][o.o.k.i.KNNSettingsTests ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] before test
  1> [2024-02-07T14:31:38,737][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] version[2.11.1-SNAPSHOT], pid[2275], build[unknown/6b1986e964d440be9137eba1413015c31c5a7752/2023-11-29T21:17:26.630182988Z], OS[Linux/6.2.0-1017-aws/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7]
  1> [2024-02-07T14:31:38,737][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] JVM home [/opt/java/openjdk-17]
  1> [2024-02-07T14:31:38,737][DEPRECATION][o.o.d.n.Node             ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
  1> [2024-02-07T14:31:38,738][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] JVM arguments [-Dgradle.dist.lib=/home/ci-runner/.gradle/wrapper/dists/gradle-7.6.1-all/942lu1p9i6mhoyzmt401s4g74/gradle-7.6.1/lib, -Dgradle.user.home=/home/ci-runner/.gradle, -Dgradle.worker.jar=/home/ci-runner/.gradle/caches/7.6.1/workerMain/gradle-worker.jar, -Dio.netty.noKeySetOptimization=true, -Dio.netty.noUnsafe=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Djava.awt.headless=true, -Djava.library.path=/home/ci-runner/k-NN/jni/release, -Djava.locale.providers=SPI,COMPAT, -Djdk.attach.allowAttachSelf=true, -Djna.nosys=true, -Dopensearch.scripting.update.ctx_in_params=false, -Dopensearch.search.rewrite_sort=true, -Dopensearch.transport.cname_in_publish_address=true, -Dorg.gradle.internal.worker.tmpdir=/home/ci-runner/k-NN/build/tmp/test/work, -Dorg.gradle.native=false, -Dtests.artifact=opensearch-knn, -Dtests.gradle=true, -Dtests.logger.level=WARN, -Dtests.security.manager=false, -Dtests.seed=1D74642C8A5A3ABE, -Dtests.task=:test, -XX:+HeapDumpOnOutOfMemoryError, -esa, --illegal-access=warn, -javaagent:../../tmp/expandedArchives/org.jacoco.agent-0.8.7.jar_3a83c50b4a016f281c4e9f3500d16b55/jacocoagent.jar=destfile=../../jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false, -XX:HeapDumpPath=/home/ci-runner/k-NN/build/heapdump, -Xms512m, -Xmx512m, -Dfile.encoding=US-ASCII, -Djava.io.tmpdir=/home/ci-runner/k-NN/build/testrun/test/temp, -Duser.country=US, -Duser.language=en, -Duser.variant, -ea]
  1> [2024-02-07T14:31:38,738][WARN ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] version [2.11.1-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
  1> [2024-02-07T14:31:38,738][INFO ][o.o.p.PluginsService     ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] no modules loaded
  1> [2024-02-07T14:31:38,739][INFO ][o.o.p.PluginsService     ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] loaded plugin [org.opensearch.knn.plugin.KNNPlugin]
  1> [2024-02-07T14:31:38,739][INFO ][o.o.p.PluginsService     ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
  1> [2024-02-07T14:31:38,739][INFO ][o.o.p.PluginsService     ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
  1> [2024-02-07T14:31:38,739][INFO ][o.o.e.ExtensionsManager  ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] ExtensionsManager initialized
  1> [2024-02-07T14:31:38,741][INFO ][o.o.e.NodeEnvironment    ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] using [1] data paths, mounts [[/ (overlay)]], net usable_space [475.7gb], net total_space [484.4gb], types [overlay]
  1> [2024-02-07T14:31:38,741][INFO ][o.o.e.NodeEnvironment    ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] heap size [512mb], compressed ordinary object pointers [true]
  1> [2024-02-07T14:31:38,743][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] node name [mock_ node], node ID [NoDIkP__T_-xKXOs_____w], cluster name [single-node-cluster-TEST_WORKER_VM=[4]-CLUSTER_SEED=[-7675565384823056753]-HASH=[1185C65F8D7]], roles [ingest, remote_cluster_client, data, cluster_manager]
  1> [2024-02-07T14:31:38,765][DEPRECATION][o.o.d.c.s.Settings       ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  1> [2024-02-07T14:31:38,785][INFO ][o.o.d.DiscoveryModule    ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] using discovery type [zen] and seed hosts providers [settings]
  1> [2024-02-07T14:31:38,821][WARN ][o.o.g.DanglingIndicesState] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
  1> [2024-02-07T14:31:38,836][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] initialized
  1> [2024-02-07T14:31:38,837][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] starting ...
  1> [2024-02-07T14:31:38,847][INFO ][o.o.t.TransportService   ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
  1> [2024-02-07T14:31:38,848][INFO ][o.o.t.TransportService   ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] Remote clusters initialized successfully.
  1> [2024-02-07T14:31:38,868][INFO ][o.o.c.c.ClusterBootstrapService] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing cluster-manager is discovered
  1> [2024-02-07T14:31:41,869][INFO ][o.o.c.c.Coordinator      ] [mock_ node] setting initial configuration to VotingConfiguration{NoDIkP__T_-xKXOs_____w}
  1> [2024-02-07T14:31:41,984][INFO ][o.o.c.s.MasterService    ] [mock_ node] elected-as-cluster-manager ([1] nodes joined)[{mock_ node}{NoDIkP__T_-xKXOs_____w}{F1vrdlQVRXyX30G9Yv94VQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{mock_ node}{NoDIkP__T_-xKXOs_____w}{F1vrdlQVRXyX30G9Yv94VQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
  1> [2024-02-07T14:31:41,994][INFO ][o.o.c.c.CoordinationState] [mock_ node] cluster UUID set to [3qM8CiSXRQ25JXFJg4_uCw]
  1> [2024-02-07T14:31:42,004][INFO ][o.o.c.s.ClusterApplierService] [mock_ node] cluster-manager node changed {previous [], current [{mock_ node}{NoDIkP__T_-xKXOs_____w}{F1vrdlQVRXyX30G9Yv94VQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
  1> [2024-02-07T14:31:42,005][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] started
  1> [2024-02-07T14:31:42,005][INFO ][o.o.d.PeerFinder         ] [mock_ node] setting findPeersInterval to [1s] as node commission status = [true] for local node [{mock_ node}{NoDIkP__T_-xKXOs_____w}{F1vrdlQVRXyX30G9Yv94VQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]
  1> [2024-02-07T14:31:42,010][INFO ][o.o.g.GatewayService     ] [mock_ node] recovered [0] indices into cluster_state
  1> [2024-02-07T14:31:42,013][INFO ][o.o.p.PluginsService     ] [mock_ node] PluginService:onIndexModule index:[myindex/nzUpSzPzTlSYdkKWI4iZnQ]
  1> [2024-02-07T14:31:42,015][INFO ][o.o.c.m.MetadataCreateIndexService] [mock_ node] [myindex] creating index, cause [api], templates [], shards [1]/[1]
  1> [2024-02-07T14:31:42,029][INFO ][o.o.p.PluginsService     ] [mock_ node] PluginService:onIndexModule index:[myindex/nzUpSzPzTlSYdkKWI4iZnQ]
  1> [2024-02-07T14:31:42,077][INFO ][o.o.p.PluginsService     ] [mock_ node] PluginService:onIndexModule index:[myindex/nzUpSzPzTlSYdkKWI4iZnQ]
  1> [2024-02-07T14:31:42,097][INFO ][o.o.p.PluginsService     ] [mock_ node] PluginService:onIndexModule index:[myindex/nzUpSzPzTlSYdkKWI4iZnQ]
  1> [2024-02-07T14:31:42,113][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] stopping ...
  1> [2024-02-07T14:31:42,117][INFO ][o.o.n.Node               ] [testFilteredSearchAdvanceSetting_whenValuesProvidedByUsers_thenValidateSameValues] stopped

Edit: I was able to repro on 2.11.1 with jdk 17 with the ubuntu setup above and the command ./gradlew test --parallel. After disabling warnings check, the tests pass.

heemin32 commented 8 months ago

Instead of removing the validation, shouldn't we just comment it out temporary and create a new issue to dig in why it is failing? I guess somehow it gets over written by other threads?

jmazanec15 commented 8 months ago

@heemin32 Do we need the validation? Removal was because we dont really need the check in our tests and it could take awhile to debug. So having it does not seem to add value

heemin32 commented 8 months ago

Okay. The warning is coming from opensearch core because if are trying to update deprecated setting? At least we need to leave a comment on why we set enableWarningCheck as false then.

junqiu-lei commented 8 months ago

@heemin32 Added comment on enableWarningCheck method https://github.com/opensearch-project/k-NN/pull/1442/commits/94aaa9024da5c7996481a2a51a38b2681d650aac

junqiu-lei commented 8 months ago

Closing this issue since https://github.com/opensearch-project/k-NN/pull/1442 got merged