search

opensearch-project / anomaly-detection-dashboards-plugin

Manage your detectors and identify atypical data in OpenSearch Dashboards
https://opensearch.org/docs/latest/monitoring-plugins/ad/index/
Apache License 2.0
30 stars 58 forks source link

[BUG] [3.0/main branch] Failing remote integ test (create detector flow) #381

Closed amitgalitz closed 1 year ago

amitgalitz commented 1 year ago

What is the bug? Our main branch has a consistently failing remote integration test. This can't be reproduced either locally or on GH runners against our 2.x branch.

The test that has failed consistently 6 times in a row on GH runner is the create detector flow.

error: AssertionError: Timed out retrying after 60000ms: Expected to find element: [data-test-subj="defineOrEditDetectorTitle"], but never found it.

This is likely due to an issue during the bulk indexing of sample data which is the only thing that happens before the line it fails on, meaning we aren't able to move to the create detector page

Additionally AD Dashboard page tests have been flaky during these re-runs on Github, 6 re-runs here: https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/actions/runs/3715398767/jobs/6377116197

When trying to reproduce the create detector flow failure locally, I was able to actually pass the line where it fails on GH runner but consistently fail on AssertionError: Timed out retrying after 60000ms: Expected <h1.euiTitle.euiTitle--large> not to exist in the DOM, but it was continuously found. This error happens when we try to move on from the first step in the create detector page to the next step of setting up the configuration so later on then the failures seen on the GH runner.

What is your host/environment?

kaituo commented 1 year ago

The error appears in 2.5 and 2.x as well.

amitgalitz commented 1 year ago

I see that this failure also occurred in windows in some of our runs, however 2.5.0 has passed all tests through jenkins. Are the failures you are seeing on windows only? Or also just other additional flaky tests you see in linux?

kaituo commented 1 year ago

Mostly on windows. But in my recent PR run, it fails on Linux as well due to times out waiting for server to start: https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/pull/399

Remote integ test on ubuntu failed with:

Run cypress-io/github-action@v2
Received 21230857 of 21230857 (100.0%), 73.1 MBs/sec
Cache Size: ~20 MB (21230857 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/5c5a8104-b187-4b74-ac18-32ba6be08b9c/cache.tzst -P -C /home/runner/work/anomaly-detection-dashboards-plugin/anomaly-detection-dashboards-plugin
Cache restored successfully
Received 205520896 of 336239223 (61.1%), 195.6 MBs/sec
Received 336239223 of 336239223 (100.0%), 172.1 MBs/sec
Cache Size: ~321 MB (336239223 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/b27dd65c-651e-4044-aad4-36a8311b3944/cache.tzst -P -C /home/runner/work/anomaly-detection-dashboards-plugin/anomaly-detection-dashboards-plugin
Cache restored successfully
/opt/hostedtoolcache/node/14.20.1/x64/bin/npm ci

> cypress@9.5.4 postinstall /home/runner/work/anomaly-detection-dashboards-plugin/anomaly-detection-dashboards-plugin/opensearch-dashboards-functional-test/node_modules/cypress
> node index.js --exec install

Note: Overriding Cypress cache directory to: ~/.cache/Cypress

      Previous installs of Cypress may not be found.

Cypress 9.5.4 is installed in /home/runner/.cache/Cypress/9.5.4

> opensearch-dashboards-functional-test@2.5.0 postinstall /home/runner/work/anomaly-detection-dashboards-plugin/anomaly-detection-dashboards-plugin/opensearch-dashboards-functional-test
> husky install

husky - Git hooks installed
added 327 packages in 5.274s
/opt/hostedtoolcache/node/14.20.1/x64/bin/npx cypress cache list
┌─────────┬───────────────────┐
│ version │ last used         │
├─────────┼───────────────────┤
│ 12.3.0  │ a few seconds ago │
├─────────┼───────────────────┤
│ 9.5.4   │ a few seconds ago │
└─────────┴───────────────────┘
Using custom test command: yarn run cypress run --env SECURITY_ENABLED=false --spec cypress/integration/plugins/anomaly-detection-dashboards-plugin/*.js
run tests command "yarn run cypress run --env SECURITY_ENABLED=false --spec cypress/integration/plugins/anomaly-detection-dashboards-plugin/*.js"
current working directory "/home/runner/work/anomaly-detection-dashboards-plugin/anomaly-detection-dashboards-plugin/opensearch-dashboards-functional-test"
/opt/hostedtoolcache/node/14.20.1/x64/bin/yarn run cypress run --env SECURITY_ENABLED=false --spec cypress/integration/plugins/anomaly-detection-dashboards-plugin/*.js
yarn run v1.21.1
$ /home/runner/work/anomaly-detection-dashboards-plugin/anomaly-detection-dashboards-plugin/opensearch-dashboards-functional-test/node_modules/.bin/cypress run --env SECURITY_ENABLED=false --spec 'cypress/integration/plugins/anomaly-detection-dashboards-plugin/*.js'
[4512:0119/200952.913642:ERROR:gpu_init.cc(453)] Passthrough is not supported, GL is swiftshader, ANGLE is 
Cypress could not verify that this server is running:

  > http://localhost:5601

We are verifying this server because it has been configured as your baseUrl.

Cypress automatically waits until your server is accessible before running tests.

We will try connecting to it 3 more times...
We will try connecting to it 2 more times...
We will try connecting to it 1 more time...

Cypress failed to verify that your server is running.

Please start this server and then run Cypress again.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: The process '/opt/hostedtoolcache/node/14.20.1/x64/bin/yarn' failed with exit code 1
kaituo commented 1 year ago

another failure on remote integ test run on Linux using main branch:

https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/actions/runs/4537480225/jobs/7995374121?pr=446

Running:  plugins/anomaly-detection-dashboards-plugin/dashboard_spec.js                   (2 of 8)

  AD Dashboard page
    1) Empty - no detector index
    2) Empty - empty detector index
    ✓ Non-empty - single running detector (16102ms)
    ✓ Redirect to create detector page (9750ms)
    ✓ Filter by detector (8992ms)
    ✓ Filter by detector state (7799ms)
    ✓ Filter by index (8264ms)
    ✓ Enter and exit full screen (7263ms)

  6 passing (3m)
  2 failing

  1) AD Dashboard page
       Empty - no detector index:
     CypressError: Timed out retrying after 60000ms: `cy.wait()` timed out waiting `60000ms` for the 1st request to the route: `getDetectors`. No request ever occurred.

https://on.cypress.io/wait
      at cypressErr (http://localhost:5601/__cypress/runner/cypress_runner.js:176389:18)
      at Object.errByPath (http://localhost:5601/__cypress/runner/cypress_runner.js:176458:10)
      at checkForXhr (http://localhost:5601/__cypress/runner/cypress_runner.js:162574:92)
      at http://localhost:5601/__cypress/runner/cypress_runner.js:162597:28
      at tryCatcher (http://localhost:5601/__cypress/runner/cypress_runner.js:13022:23)
      at Function.Promise.attempt.Promise.try (http://localhost:5601/__cypress/runner/cypress_runner.js:10296:29)
      at whenStable (http://localhost:5601/__cypress/runner/cypress_runner.js:168808:63)
      at http://localhost:5601/__cypress/runner/cypress_runner.js:168305:14
      at tryCatcher (http://localhost:5601/__cypress/runner/cypress_runner.js:13022:23)
      at Promise._settlePromiseFromHandler (http://localhost:5601/__cypress/runner/cypress_runner.js:10957:31)
      at Promise._settlePromise (http://localhost:5601/__cypress/runner/cypress_runner.js:11014:18)
      at Promise._settlePromise0 (http://localhost:5601/__cypress/runner/cypress_runner.js:11059:10)
      at Promise._settlePromises (http://localhost:5601/__cypress/runner/cypress_runner.js:11139:18)
      at Promise._fulfill (http://localhost:5601/__cypress/runner/cypress_runner.js:11083:18)
      at http://localhost:5601/__cypress/runner/cypress_runner.js:12697:46
  From Your Spec Code:
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/support/index.js:1325:6)

  2) AD Dashboard page
       Empty - empty detector index:
     CypressError: Timed out retrying after 60000ms: `cy.wait()` timed out waiting `60000ms` for the 1st request to the route: `getDetectors`. No request ever occurred.

https://on.cypress.io/wait
      at cypressErr (http://localhost:5601/__cypress/runner/cypress_runner.js:176389:18)
      at Object.errByPath (http://localhost:5601/__cypress/runner/cypress_runner.js:176458:10)
      at checkForXhr (http://localhost:5601/__cypress/runner/cypress_runner.js:162574:92)
      at http://localhost:5601/__cypress/runner/cypress_runner.js:162597:28
      at tryCatcher (http://localhost:5601/__cypress/runner/cypress_runner.js:13022:23)
      at Function.Promise.attempt.Promise.try (http://localhost:5601/__cypress/runner/cypress_runner.js:10296:29)
      at whenStable (http://localhost:5601/__cypress/runner/cypress_runner.js:168808:63)
      at http://localhost:5601/__cypress/runner/cypress_runner.js:168305:14
      at tryCatcher (http://localhost:5601/__cypress/runner/cypress_runner.js:13022:23)
      at Promise._settlePromiseFromHandler (http://localhost:5601/__cypress/runner/cypress_runner.js:10957:31)
      at Promise._settlePromise (http://localhost:5601/__cypress/runner/cypress_runner.js:11014:18)
      at Promise._settlePromise0 (http://localhost:5601/__cypress/runner/cypress_runner.js:11059:10)
      at Promise._settlePromises (http://localhost:5601/__cypress/runner/cypress_runner.js:11139:18)
      at Promise._fulfill (http://localhost:5601/__cypress/runner/cypress_runner.js:11083:18)
      at http://localhost:5601/__cypress/runner/cypress_runner.js:12697:46
  From Your Spec Code:
      at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/support/index.js:1325:6)
ohltyler commented 1 year ago

Current status:

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  plugins/anomaly-detection-dashboard      01:00        1        1        -        -        - │
  │    s-plugin/create_detector_spec.js                                                            │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  plugins/anomaly-detection-dashboard      01:08        8        8        -        -        - │
  │    s-plugin/dashboard_spec.js                                                                  │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  plugins/anomaly-detection-dashboard      01:34        5        -        1        -        4 │
  │    s-plugin/detector_configuration_spe                                                         │
  │    c.js                                                                                        │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  plugins/anomaly-detection-dashboard      01:31       11       11        -        -        - │
  │    s-plugin/detector_list_spec.js                                                              │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  plugins/anomaly-detection-dashboard      02:14        6        6        -        -        - │
  │    s-plugin/historical_analysis_spec.j                                                         │
  │    s                                                                                           │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  plugins/anomaly-detection-dashboard      00:48        4        4        -        -        - │
  │    s-plugin/overview_spec.js                                                                   │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  plugins/anomaly-detection-dashboard      00:44        2        2        -        -        - │
  │    s-plugin/real_time_results_spec.js                                                          │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  plugins/anomaly-detection-dashboard      01:12        3        3        -        -        - │
  │    s-plugin/sample_detector_spec.js                                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 8 failed (13%)                      10:15       40       35        1        -        4  
  0 passing (2m)
  1 failing

  1) Detector configuration page
       "before all" hook for "Redirect to edit detector settings from button":
     AssertionError: Timed out retrying after 60000ms: Expected to find element: `[data-test-subj="viewSampleDetectorLink"]`, but never found it.
ohltyler commented 1 year ago

The failures seem to be happening in windows envs every time, where the test cluster looks to be extremely unstable based on the time elapsed for running even very basic tests. Opened an issue in FTRepo for more tracking. The only failures I've seen lately are just due to this. All of the tests run by infra during release time for the last 2 releases haven't seen any AD failures, since those clusters are stable, running in the more-resourced jenkins hosts.

Opened issue here: https://github.com/opensearch-project/opensearch-dashboards-functional-test/issues/615

Will just continue to track there since there's no urgent failing / flaky tests on our end since it is strictly test env issues.