search

opensearch-project / opensearch-plugins

For all things OpenSearch plugins. You want to install, or develop a plugin? You've come to the right place.
Apache License 2.0
51 stars 61 forks source link

[PROPOSAL] Add flaky test retry #139

Open dblock opened 2 years ago

dblock commented 2 years ago

What kind of business use case are you trying to solve? What are your requirements? I want to be able to rely on CI when contributing to all plugin projects.

What is the problem? What is preventing you from meeting the requirements? In a trivial change in https://github.com/opensearch-project/.github/issues/64 I made 20 pull requests with a trivial .md file change. 10/20 have failed in CI, anecdotally because of flakey tests.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation? Introduce a small number of retries as standard procedure to tests, similar to https://github.com/opensearch-project/OpenSearch/issues/2547, implemented in https://github.com/opensearch-project/OpenSearch/pull/2638. While this doesn't fix flakey tests, it decreases the noise to a level where the worst offenders can be fixed instead of having a 50% failure rate.

What are remaining open questions? We know how to do this for OpenSearch plugins with Gradle, but we need another mechanism for OpenSearch Dashboards.

saratvemulapalli commented 2 years ago

@dblock looks like this slipped through the cracks. Do you want to open up issues on all the repos and label them for next release?

dblock commented 2 years ago

@saratvemulapalli if you want to take this campaign up go for it!

ohltyler commented 1 year ago

Similar comment as https://github.com/opensearch-project/opensearch-plugins/issues/138#issuecomment-1372798845