stephen-crawford
commented
1 year ago I am going to go ahead and get this ball rolling.
Some things I would be interested in seeing are:
-
New tests auto-run: Keep the existing gradle check, but add a workflow that automatically scanned for new test files (gradle + GitHub should be able to do this) and then runs those tests separately. This would let you know whether there was an issue with the test you just wrote or something related to how the tests interact with the older test suite. @cwperks wrote a file that does something along these lines over in the Identity project, and it has worked well for us. I think taking Craig's idea and going one step further could be helpful for a lot of repos.
-
Dashboards auto-push to functional repo: I have heard from some dashboards-minded folks that the functional tests repo does not always get individual repos' tests. I am not much of a dashboards expert myself so do not know what is required here, but it sounds like adding a check to make sure that all tests are added to the functional testing repo would be helpful.
-
Documentation check: Something else that I heard was that there are some challenges keeping documentation relevant and up-to-date. I think it could be helpful to add an auto-check like there is for the
CHANGELOGthat checked for documentation updates or required you to override it stating that your change only modified code irrelevant to feature execution. I.e. you updated a dependency or added an extra test.
peternied
davidlago
joshuali925
TL;DR: What do you want the organization to use as testing requirements? What certifies something is "done?"
What are you proposing?
A major topic discussed at the milestone meetings hosted by @CEHENKLE, has been establishing a universal quality bar. An actionable mechanism for the this is using GitHub actions and checks to assert changes meet expectations.
This issue should be used as a discussion board for what these expectations are. Everyone across the @opensearch-project, is encouraged to contribute as this is a change that will affect us all. The final result of this discussion will be a "universal" set of expectations which all repositories will employ. This means that maintainers will have an established quality bar for merging changes, and that contributors can know all requirements before they start working.
What users have asked for this feature?
The milestone group worked with numerous repositories to learn their pain points--the things that were preventing them from releasing on time and making a great product. One of the consistent patterns was a lack of established expectations. One group wanted changes to have "a, b, and c," while another wanted "x, y, and z" This proposal seeks to establish a set of accepted expectations across all repositories.
What is the contributor experience going to be?
Under the new system, a contributor will be able to reference the
CONTRIBUTING.mddocument of any @opensearch-project repository and see the repository's expectations. Most of these expectations will be standardized across the organization but there may be additional requirements depending on each repositories use case. For example, the Security repository may require that the plugin install workflow passes but this would not be required for the whole organization.This discussion establishes a base quality bar across the organization. If any repository wants to add further requirements specific to them that is encouraged. They will need to list the additional requirements alongside the standardized ones in their
CONTRIBUTING.md.How does this impact flaky tests?
Currently, we are trying to reduce the number of flaky tests which exist in the project. Unfortunately, this is easier said than done and the process is ongoing. Part of this discussion should be to establish expectations for handling existing flaky tests and it is likely that the new quality bar will expect no more flaky tests are introduced. A method of doing this is running all new tests in a separate workflow which is isolated from the older tests. This prevents any currently flaky tests from impacting newly added changes.
What will it take to implement?
In order for repositories to adopt the new quality bar, they should update their
CONTRIBUTING.mddocument and their testing frameworks. Since the expectations are being standardized, it should be relatively easy to port between repositories. If tests are written as GitHub Actions they can likely be directly copied with minimal changes.Examples of some of the types of tests that may be helpful can be found in the identity project which has workflows for testing a wide range of functions on commit. There may be many more tests that should be mentioned however, and expectations can also include things like updated developer guides.
Any remaining open questions?
We are still trying to determine what these expectations should be so it is important that all opinions are heard. If you have any comments, concerns, or suggestions please share them below.