[META] CI Improvement Plan

[seanneumann]

We'd like to improve our CI for OpenSearch Dashboards. Creating this issue to start forming the plan.

Initial AI: baseline metrics to measure progress against and establish goal.

CI Labelled Issues

• Standardized plugin CI
  • Problem
  • Each plugin (especially in the OpenSearch project org) has defined their own GitHub CI workflows which vary from repo to repo. Sometimes a repo will lack verification within release builds which then leads to bugs only being caught close to release or do not utilize available scripts that can make their test runs more consistent or faster.
  • Solution
  • Provide scripts (potentially housed and imported by the SDK or within the plugin helpers package) to allow for plugins to build and verify to standardized what is being executed and called from CI workflows.
  • Tasks
  • https://github.com/opensearch-project/OpenSearch-Dashboards/issues/3918
• Reduce monolith script
  • Problem
  • Build and test CI workflow is massive and there is a lot of replication between the steps. It makes it difficult to contribute, improve, and fix whenever there are issues.
  • Solution
  • Breakdown the script into components and templates
  • Tasks
  • https://github.com/opensearch-project/OpenSearch-Dashboards/issues/3603
  • https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4031
  • https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4032
• Reduce flakiness of tests
• Ensure timeouts of tests
• Migrate to Selenium tests to Cypress tests
  • Screenshot/video upload to PR (already occurring but make it a comment for ease of access of maintainers to quickly review any potential UI regressions)
• Failed backport label automation
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4129

[kavilla]

Device matrix testing https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4803