[BUG] Tracking issue for known flaky tests

AWSHurneyt commented 1 year ago

Please add more flaky tests to this issue as they're discovered; or notes from investigations into the causes of the flaky tests. Reducing flakiness will increase confidence in our automated testing, and reduce the amount of time needed to merge changes as rerunning cypress tests is a time-consuming process.

List of known flaky tests

bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor
1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.
```
CypressError: Timed out retrying after 10000ms: `cy.select()` failed because it could not find a single `<option>` with value or text matching: `avg_products_base_price`
```
~~query_level_monitor_spec.js > schedule component displays as intended > for a cron expression schedule~~
1. ~~This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.~~
2. ~~This flaky behavior is also rarely seen impacting the query_level_monitor_spec.js > schedule component displays as intended > for an interval schedule test.~~
```
AssertionError: Timed out retrying after 20000ms: Expected to find element: `[data-test-subj="frequency_field"]`, but never found it.
at Context.eval (https://search-t-nobody-avjv-e2ealerting-3rb2m37iqureafmtutcxcgypem.eu-west-1.es-staging.amazonaws.com/__cypress/tests?p=cypress/integration/plugins/alerting-dashboards-plugin/query_level_monitor_spec.js:580:10)
```
bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor seems to be flaky when executed by the 1.3 functional test repo branch. It has failed with the error below.
1. ~~This seems to relate to the use of {enter} when using cy.type() here. Cypress fails to execute the {enter} press.~~
  1. ~~The screenshot below helps illustrate this; the enter key press doesn't execute, so the UI doesn't detect that a Time Field has been defined which prevents the subsequent UI from rendering.~~
  2. ~~A potential solution may involve changing a .blur() call at the end of .type() calls that use {enter}.~~
```
AssertionError: Timed out retrying: Expected to find element: `[data-test-subj="addMetricButton"]`, but never found it.
```
cluster_metrics_monitor_spec.js > can be created > for the Cluster Health API seems to occasionally fail when the API is selected from the dropdown before page finishes loading. This causes the selection to reset which causes the test to fail as it expects to see UI elements for the selection. See image screenshot below

~~monitors_dashboard_spec.js > Displays expected number of alerts occasionally fails with the following error (example). May be related to a delay in alert ingestion.~~

AssertionError: Timed out retrying after 20000ms: Expected to find element: `:contains(monitors_dashboard_cypress_query_level)`, but never found it. Queried from element: <tr.euiTableRow>

composite_level_monitor_spec.js > can be edited > by visual editor occasionally fails with the following error (example). Adding a short wait after this cy.visit call (link) helps to alleviate the flakiness; but a more permanent solution should be implemented.

1) CompositeLevelMonitor
   can be edited
     by visual editor:
 AssertionError: Timed out retrying after 10000ms: Expected to find element: `[data-test-subj="monitors_list_2"]`, but never found it.
  at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/composite_level_monitor_spec.js:339:10)

AWSHurneyt commented 1 year ago

With the changes made to query_level_monitor_spec.js in PR https://github.com/opensearch-project/opensearch-dashboards-functional-test/pull/622, the second test in the list doesn't appear to be failing periodically anymore. I've scratched it off the list, but we will continue to monitor it.

AWSHurneyt commented 1 year ago

Reopening issue as flaky test 5 still seems to be an issue as seen in PR https://github.com/opensearch-project/alerting-dashboards-plugin/pull/767

AWSHurneyt commented 1 year ago

wang-chenxi commented 1 year ago

Hello, in the description it mentioned bucket-level monitor twice, i wonder if one of them is compositeLevelMonitor? i got both bucket-level and composite-level failed for my PR's E2E test. https://github.com/opensearch-project/alerting-dashboards-plugin/actions/runs/6778149217/job/18423118183?pr=794#step:13:307

AWSHurneyt commented 1 year ago

PR https://github.com/opensearch-project/opensearch-dashboards-functional-test/pull/953 should help address flakiness of test monitors_dashboard_spec.js > Displays expected number of alerts when the tests are executed in the FTR for releases.

AWSHurneyt commented 1 year ago

@wang-chenxi bucket level is mentioned twice in the description because there were a few different flaky observations in that suite. Those bucket level entries have been tracked as flaky since before the release of composite monitors.

AWSHurneyt commented 3 months ago

jowg-amazon commented 2 weeks ago

Adding another known flakey test:

AcknowledgeAlertsModal > "before all" hook for "Acknowledge button disabled when more than 1 trigger selected"

Error:

1) AcknowledgeAlertsModal
       "before all" hook for "Acknowledge button disabled when more than 1 trigger selected":
     CypressError: `cy.request()` timed out waiting `30000ms` for a response from your server.

CI run link: https://github.com/opensearch-project/alerting-dashboards-plugin/actions/runs/11584512757/job/32251792002

opensearch-project / alerting-dashboards-plugin

[BUG] Tracking issue for known flaky tests #415