search

opensearch-project / alerting-dashboards-plugin

📟 Manage your monitors, alerts and notifications in OpenSearch Dashboards
https://opensearch.org/docs/latest/monitoring-plugins/alerting/index/
Apache License 2.0
26 stars 87 forks source link

[BUG] Tracking issue for known flaky tests #415

Open AWSHurneyt opened 1 year ago

AWSHurneyt commented 1 year ago

Please add more flaky tests to this issue as they're discovered; or notes from investigations into the causes of the flaky tests. Reducing flakiness will increase confidence in our automated testing, and reduce the amount of time needed to merge changes as rerunning cypress tests is a time-consuming process.

List of known flaky tests

  1. bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor

    1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails. 
      CypressError: Timed out retrying after 10000ms: `cy.select()` failed because it could not find a single `<option>` with value or text matching: `avg_products_base_price`

  2. query_level_monitor_spec.js > schedule component displays as intended > for a cron expression schedule

    1. This test case seems to be much less flaky when executed locally, but will occasionally fail when executed by github actions. The following error is observed when the test fails.
    2. This flaky behavior is also rarely seen impacting the query_level_monitor_spec.js > schedule component displays as intended > for an interval schedule test. 
      AssertionError: Timed out retrying after 20000ms: Expected to find element: `[data-test-subj="frequency_field"]`, but never found it.
at Context.eval (https://search-t-nobody-avjv-e2ealerting-3rb2m37iqureafmtutcxcgypem.eu-west-1.es-staging.amazonaws.com/__cypress/tests?p=cypress/integration/plugins/alerting-dashboards-plugin/query_level_monitor_spec.js:580:10)

  3. bucket_level_monitor_spec.js > Bucket-Level Monitors > can be created > by visual editor seems to be flaky when executed by the 1.3 functional test repo branch. It has failed with the error below.

    1. This seems to relate to the use of {enter} when using cy.type() here. Cypress fails to execute the {enter} press.
      1. The screenshot below helps illustrate this; the enter key press doesn't execute, so the UI doesn't detect that a Time Field has been defined which prevents the subsequent UI from rendering.
      2. A potential solution may involve changing a .blur() call at the end of .type() calls that use {enter}. 
        AssertionError: Timed out retrying: Expected to find element: `[data-test-subj="addMetricButton"]`, but never found it.
        Screenshot 2023-03-27 at 1 06 47 PM

  4. cluster_metrics_monitor_spec.js > can be created > for the Cluster Health API seems to occasionally fail when the API is selected from the dropdown before page finishes loading. This causes the selection to reset which causes the test to fail as it expects to see UI elements for the selection. See image screenshot below

    Screenshot 2023-05-18 at 11 48 04 AM

  5. monitors_dashboard_spec.js > Displays expected number of alerts occasionally fails with the following error (example). May be related to a delay in alert ingestion.

    AssertionError: Timed out retrying after 20000ms: Expected to find element: `:contains(monitors_dashboard_cypress_query_level)`, but never found it. Queried from element: <tr.euiTableRow>
AWSHurneyt commented 1 year ago

With the changes made to query_level_monitor_spec.js in PR https://github.com/opensearch-project/opensearch-dashboards-functional-test/pull/622, the second test in the list doesn't appear to be failing periodically anymore. I've scratched it off the list, but we will continue to monitor it.

AWSHurneyt commented 11 months ago

Reopening issue as flaky test 5 still seems to be an issue as seen in PR https://github.com/opensearch-project/alerting-dashboards-plugin/pull/767

AWSHurneyt commented 11 months ago

Related issue https://github.com/opensearch-project/alerting-dashboards-plugin/issues/762

wang-chenxi commented 10 months ago

Hello, in the description it mentioned bucket-level monitor twice, i wonder if one of them is compositeLevelMonitor? i got both bucket-level and composite-level failed for my PR's E2E test. https://github.com/opensearch-project/alerting-dashboards-plugin/actions/runs/6778149217/job/18423118183?pr=794#step:13:307

AWSHurneyt commented 10 months ago

PR https://github.com/opensearch-project/opensearch-dashboards-functional-test/pull/953 should help address flakiness of test monitors_dashboard_spec.js > Displays expected number of alerts when the tests are executed in the FTR for releases.

AWSHurneyt commented 10 months ago

@wang-chenxi bucket level is mentioned twice in the description because there were a few different flaky observations in that suite. Those bucket level entries have been tracked as flaky since before the release of composite monitors.

AWSHurneyt commented 1 month ago

Related issue https://github.com/opensearch-project/opensearch-dashboards-functional-test/issues/1464