search

opensearch-project / flow-framework

OpenSearch plugin that enables builders to innovate AI apps on OpenSearch
Apache License 2.0
33 stars 36 forks source link

[AUTOCUT] Integration Test failed for flow-framework: 2.17.0 #856

Closed opensearch-ci-bot closed 2 months ago

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8609/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10268/linux/arm64/tar/test-results/8609/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8610/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10268/linux/x64/tar/test-results/8610/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8611/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/arm64/tar/test-results/8611/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8614/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/x64/tar/test-results/8614/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8613/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/x64/rpm/test-results/8613/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8616/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/arm64/rpm/test-results/8616/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8617/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/x64/deb/test-results/8617/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8615/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/arm64/deb/test-results/8615/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8619/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/tar/test-results/8619/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8622/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/deb/test-results/8622/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8621/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/rpm/test-results/8621/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8624/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/deb/test-results/8624/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8620/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/tar/test-results/8620/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8623/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/rpm/test-results/8623/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8627/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/tar/test-results/8627/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8626/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/windows/x64/zip/test-results/8626/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8630/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/tar/test-results/8630/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

dbwiddis commented 2 months ago

src/test/java/org/opensearch/flowframework/workflow/RegisterLocalPretrainedModelStepTests.java:140: error: no suitable constructor found for MLTask(String,String,,,MLTaskState,,,,,,,,,boolean)

Looks like https://github.com/opensearch-project/ml-commons/pull/2825 is a breaking change in a minor version bump. @rbhavna is this intentional?

CC @Zhangxunmt @ylwu-amzn @minalsha

dbwiddis commented 2 months ago

Closing as this has passed on all platforms.

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8670/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10289/windows/x64/zip/test-results/8670/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

dbwiddis commented 2 months ago

Failure looks flaky, has passed multiple previous times.

opensearch-ci-bot commented 2 months ago

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8673/display/redirect

Test-report manifest:
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10289/windows/x64/zip/test-results/8673/integ-test/test-report.yml

_Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here._

dbwiddis commented 2 months ago

Passed on retry: https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/8674/pipeline

gaiksaya commented 2 months ago

@dbwiddis For future releases can you check why flow-framework is being flaky for windows platform? Had to run multiple times to get it to pass. Thank you!

dbwiddis commented 2 months ago

@dbwiddis For future releases can you check why flow-framework is being flaky for windows platform? Had to run multiple times to get it to pass. Thank you!

Our integ tests load local models into memory, which often causes memory issues in CI test environments. It's much more common on macOS github runners, but occasionally happens for WIndows.

We could probably try to refactor and do more cleanup between test runs... will create an issue.

dbwiddis commented 2 months ago

(FYI, ML Commons integ tests often trip memory circuit breakers for similar reasons. Any suggestions welcome.)

ylwu-amzn commented 2 months ago

Tune the memory circuit breaker setting ?

dbwiddis commented 2 months ago

OK, so it turns out this isn't memory related. I suspect a race condition somewhere due to how the test is written. Opened https://github.com/opensearch-project/flow-framework/issues/870, we'll fix before 2.18.

CC @gaiksaya @minalsha