Setup stricter rules in our CI to prevent introduction of new flaky tests

hjetpoluru commented 5 months ago

What is this about?

Feedback from David Walsh

I guess making the driver APIs stricter?

Feedback from Legobeat

sounds good, but maybe just removing retries for a while will be a great start

Scenario

No response

Design

No response

Technical Details

No response

Threat Modeling Framework

No response

Acceptance Criteria

No response

Stakeholder review needed before the work gets merged

[ ] Engineering (needed in most cases)
[ ] Design
[ ] Product
[ ] QA (automation tests are required to pass before merging PRs but not all changes are covered by automation tests - please review if QA is needed beyond automation tests)
[ ] Security
[ ] Legal
[ ] Marketing
[ ] Management (please specify)
[ ] Other (please specify)

References

No response

chloeYue commented 3 months ago

Close this ticket as we have introduced quality gate here: https://github.com/MetaMask/metamask-extension/pull/24556

hjetpoluru commented 3 months ago

@chloeYue, This ticket was created to disable the retry button in CI, and to ensure that the testing driver API cannot be modified easily. Neither task has been completed yet. The quality gate you mentioned that newly created tests should be checked for flakiness that is implemented and close here. https://github.com/MetaMask/metamask-extension/issues/24009

MetaMask / metamask-extension