Open manasvinibs opened 11 months ago
For Pre-existing test logic
, I guess after this issue gets done, the pre-existing test logic should goes to respective plugin repos right?
As we have security plugin, how will all the test cases be triggered under security mode and unsecurity mode?
For
Pre-existing test logic
, I guess after this issue gets done, the pre-existing test logic should goes to respective plugin repos right? Eventually yes but not necessarily part of this issue. This RFC mainly focuses on giving ability for the tests to reside where code exist and still be part of the Functional test repo's release workflow. Pre-existing logic can still remain in FT repo as orchestrator will be able to support both current and future test runs.
@manasvinibs How will this change impact the current Jenkins integration test workflows?
@manasvinibs How will this change impact the current Jenkins integration test workflows?
This change is incorporated by integrating it within integtest.sh
script into the Functional Test (FT) Repo. The integtest script is currently being invoked by the existing Jenkins integration test workflow during the execution of Cypress tests within the Functional test repo. Consequently, the orchestrator script is also triggered as part of the workflow, initiating remote workflow tests.
For the initial iteration, the objective is to seamlessly integrate the script into the process and conduct a dry run to validate its functionality. This dry run will serve as a preliminary test to ensure the smooth execution of the script within the Jenkins job.
I will be doing POC on moving future tests out of FT repo. I will be eager for feedback now and on the first PR for this.
Problem:
Today we have Functional test repo which hosts integration tests for OpenSearch Dashboards along with tests for plugin components. The existing system we employ involves a separation between the tests and test infrastructure (located in a functional test repo) and the primary repository containing our application code (OpenSearch Dashboards). This separation was initially implemented to centralize the test architecture in a single location, thereby allowing Dashboards, other plugin repositories, and hosted services to leverage a complete test library for their testing needs. However, this setup introduces an unintended challenge known as the "chicken and egg" problem. This problem arises because tests and code changes cannot occur simultaneously. Instead, one must precede the other, which can disrupt the continuous integration (CI) testing process of the other component.
GOAL:
* Note: Infra/Build team will continue to rely on Functional Test repo as-is. They can opt into running it better if they want. But ideally it can be ran without any changes on their side, and solves the problems we see today.
[Proposal] Test orchestration and execution across multiple repositories
To address the "chicken and egg" problem and the associated issues of overwriting global Cypress functions, we can use test-library, available for component repos and functional test repo to write in-house tests. Here's a proposed design:
Pros:
Cons:
Changes to existing Functional Test Repo:
Reference doc: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/3728
To ensure that the existing Functional Test Repo can run continuous integration (CI) checks as part of the release workflow on all the plugin components' tests in one place, we can establish a CI/CD workflow that coordinates the execution of tests across all repositories.
https://github.com/opensearch-project/opensearch-dashboards-test-library
Please add your suggestion in the comments below.