Add test retry logic for flaky tests

[mtrezza]

New Feature / Enhancement Checklist

• [x] I am not disclosing a vulnerability.
• [x] I am not just asking a question.
• [x] I have searched through existing issues.

Current Limitation

There are still some flaky tests;

• requires more effort to merge a PR
• shows ci: failed labels on README

Feature / Enhancement Description

Add a test-retry logic on jasmine-level (not GitHub workflow level) specifically for flaky tests.

1. Implement logic to retry specific tests if they fail, in the style of the existing custom test specifiers: https://github.com/parse-community/parse-server/blob/1850be45b17abdccae129811157ddef6feb52a34/spec/helper.js#L422
2. Identify flaky tests
3. Use retry-test-specifier for flaky tests

Alternatives / Workarounds

Manually re-run tests (status quo)

References

• https://github.com/parse-community/parse-server/issues/8613
• https://github.com/parse-community/parse-server/issues/7180
• https://github.com/parse-community/parse-server/issues/8184
• https://gist.github.com/joeyparrish/c8b020a33095ea7cf191c06f4339a4dc

[parse-github-assistant[bot]]

Thanks for opening this issue!

• 🎉 We are excited about your ideas for improvement!

[dplewis]

@mtrezza I was just looking into doing something like this. Using the test name as an identifier would be the easiest way to go.

[mtrezza]

There may be ways; I've looked into this quite extensively with jasmine but I couldn't find a way given its architecture. Hence I published https://github.com/parse-community/parse-server/pull/8655 to document my findings, closed it and kept this issue open, in case someone finds a way in the future. I think I've spent more than a day on it. The challenge is to build something on top of, or integrated into jasmine, so we have the test stats, etc. It may not be difficult to re-run a test, but I couldn't find a way to do that so that jasmine understands what's happening.

Another approach that came to my mind was to maybe use the GitHub CI to automatically create a test run per spec file. So that instead of re-running all tests when one fails, only the spec file of the failing test needs to re-run. But this'd need to consider coverage reporting, not spamming our CI dashboard, etc.

[parseplatformorg]

🎉 This change has been released in version 7.3.0-alpha.7