search

opensearch-project / neural-search

Plugin that adds dense neural retrieval into the OpenSearch ecosytem
Apache License 2.0
62 stars 65 forks source link

[AUTOCUT] Integration Test failed for neural-search: 2.7.0 tar distribution #157

Closed opensearch-ci-bot closed 1 year ago

opensearch-ci-bot commented 1 year ago

The integration test failed at distribution level for component neural-search
Version: 2.7.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/4534/display/redirect

Steps to reproduce: See https://github.com/opensearch-project/opensearch-build/tree/main/src/test_workflow#integration-tests
 Access cluster logs:
- With security (if applicable)
- Without security (if applicable)

Note: All in one test report manifest with all the details coming soon. See https://github.com/opensearch-project/opensearch-build/issues/1274

naveentatikonda commented 1 year ago

@zelinh The build is still failing with the same ClusterCreationException

File "./src/run_integ_test.py", line 36, in <module>

    sys.exit(main())

  File "./src/run_integ_test.py", line 25, in main

    all_results = IntegTestRunners.from_test_manifest(args, test_manifest).run()

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/integ_test_runner.py", line 47, in run

    test_results = test_suite.execute_tests()

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/integ_test_suite_opensearch.py", line 51, in execute_tests

    status = self.__setup_cluster_and_execute_test_config(config)

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/integ_test_suite_opensearch.py", line 86, in __setup_cluster_and_execute_test_config

    self.test_recorder

  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__

    return next(self.gen)

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/topology.py", line 50, in create

    yield topology.create_clusters()

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/topology.py", line 74, in create_clusters

    self.work_dir

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/topology.py", line 176, in __init__

    current_port

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/topology.py", line 123, in __init__

    cluster_port)

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/test_cluster.py", line 69, in create_cluster

    cluster.start()

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/test_cluster.py", line 81, in start

    service.wait_for_service()

  File "/var/jenkins/workspace/integ-test@2/src/test_workflow/integ_test/service.py", line 134, in wait_for_service

    raise ClusterCreationException("Cluster is not available after 10 attempts")

test_workflow.integ_test.service.ClusterCreationException: Cluster is not available after 10 attempts

script returned exit code 1
zelinh commented 1 year ago

@naveentatikonda I don't actually see this error. Looks like the test itself for x64 tarball failed. 

Suite: Test class org.opensearch.neuralsearch.query.NeuralQueryIT

  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.query.NeuralQueryIT.testBoostQuery" -Dtests.seed=3B6765EF713D5D62 -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Europe/Mariehamn -Druntime.java=17

  2> org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200/], URI [/test-neural-basic-index/_search?size=1], status line [HTTP/1.1 400 Bad Request]

    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"model not deployed"}],"type":"illegal_argument_exception","reason":"model not deployed"},"status":400}

        at __randomizedtesting.SeedInfo.seed([3B6765EF713D5D62:A279772AF40C0A85]:0)

        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:292)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:263)

        at app//org.opensearch.neuralsearch.query.NeuralQueryIT.testBoostQuery(NeuralQueryIT.java:115)

  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.query.NeuralQueryIT.testFilterQuery" -Dtests.seed=3B6765EF713D5D62 -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Europe/Mariehamn -Druntime.java=17

  2> org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200/], URI [/test-neural-multi-doc-index/_search?size=3], status line [HTTP/1.1 400 Bad Request]

    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"model not deployed"}],"type":"illegal_argument_exception","reason":"model not deployed"},"status":400}

        at __randomizedtesting.SeedInfo.seed([3B6765EF713D5D62:DE6B04E2A22FD5E5]:0)

        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:292)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:263)

        at app//org.opensearch.neuralsearch.query.NeuralQueryIT.testFilterQuery(NeuralQueryIT.java:341)

  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.query.NeuralQueryIT.testBooleanQuery_withMultipleNeuralQueries" -Dtests.seed=3B6765EF713D5D62 -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Europe/Mariehamn -Druntime.java=17

  2> org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200/], URI [/test-neural-multi-vector-field-index/_search?size=1], status line [HTTP/1.1 400 Bad Request]

    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"model not deployed"}],"type":"illegal_argument_exception","reason":"model not deployed"},"status":400}

        at __randomizedtesting.SeedInfo.seed([3B6765EF713D5D62:429F7D9D7B97D1BD]:0)

        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:292)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:263)

        at app//org.opensearch.neuralsearch.query.NeuralQueryIT.testBooleanQuery_withMultipleNeuralQueries(NeuralQueryIT.java:213)

  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.query.NeuralQueryIT.testBooleanQuery_withNeuralAndBM25Queries" -Dtests.seed=3B6765EF713D5D62 -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Europe/Mariehamn -Druntime.java=17

  2> org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200/], URI [/test-neural-text-and-vector-field-index/_search?size=1], status line [HTTP/1.1 400 Bad Request]

    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"model not deployed"}],"type":"illegal_argument_exception","reason":"model not deployed"},"status":400}

        at __randomizedtesting.SeedInfo.seed([3B6765EF713D5D62:4099B07B01F06046]:0)

        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:292)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:263)

        at app//org.opensearch.neuralsearch.query.NeuralQueryIT.testBooleanQuery_withNeuralAndBM25Queries(NeuralQueryIT.java:262)

  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.query.NeuralQueryIT.testNestedQuery" -Dtests.seed=3B6765EF713D5D62 -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Europe/Mariehamn -Druntime.java=17

  2> org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200/], URI [/test-neural-nested-index/_search?size=1], status line [HTTP/1.1 400 Bad Request]

    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"model not deployed"}],"type":"illegal_argument_exception","reason":"model not deployed"},"status":400}

        at __randomizedtesting.SeedInfo.seed([3B6765EF713D5D62:5B1F7638C00FD868]:0)

        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:292)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:263)

        at app//org.opensearch.neuralsearch.query.NeuralQueryIT.testNestedQuery(NeuralQueryIT.java:301)

  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.neuralsearch.query.NeuralQueryIT.testRescoreQuery" -Dtests.seed=3B6765EF713D5D62 -Dtests.security.manager=false -Dtests.locale=tr-TR -Dtests.timezone=Europe/Mariehamn -Druntime.java=17

  2> org.opensearch.client.ResponseException: method [POST], host [https://localhost:9200/], URI [/test-neural-basic-index/_search?size=1], status line [HTTP/1.1 400 Bad Request]

    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"model not deployed"}],"type":"illegal_argument_exception","reason":"model not deployed"},"status":400}

        at __randomizedtesting.SeedInfo.seed([3B6765EF713D5D62:10C1CEB43B0CEEC0]:0)

        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:375)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:345)

        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)

        at app//org.opensearch.neuralsearch.common.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:292)

        at app//org.opensearch.neuralsearch.query.NeuralQueryIT.testRescoreQuery(NeuralQueryIT.java:156)

  2> NOTE: leaving temporary files on disk at: /tmp/tmpfbg1p31v/neural-search/build/testrun/integTest/temp/org.opensearch.neuralsearch.query.NeuralQueryIT_3B6765EF713D5D62-001

  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=1471, maxMBSortInHeap=6.252906114386439, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=tr-TR, timezone=Europe/Mariehamn

  2> NOTE: Linux 5.10.176-157.645.amzn2.x86_64 amd64/Eclipse Adoptium 17.0.3 (64-bit)/cpus=16,threads=1,free=495656056,total=536870912

  2> NOTE: All tests run in this JVM: [NeuralSearchIT, ValidateDependentPluginInstallationIT, TextEmbeddingProcessorIT, NeuralQueryIT]

12 tests completed, 6 failed

You may find this link helpful. https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/4536/pipeline/108

naveentatikonda commented 1 year ago

@naveentatikonda I don't actually see this error. Looks like the test itself for x64 tarball failed.

The error I posted above was from the most recent build at that point of time. Looking into the issue which you shared above

jmazanec15 commented 1 year ago

Should be resolved with #159

naveentatikonda commented 1 year ago

Most recent build is successful. https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/4572/pipeline/

So, closing this issue.