Browser tests have their own built-in retry mechanism, but this is not sufficient on its own. We had to rebuild often enough, that we ended up adding an automated retry in CI for the entire browser test stage, including a notification whenever it's triggered. In the last 3 months we got ~160 such notifications (Slack search: "test-browser" "was retried" "times on" in:#devops-bot-dev from:@Jenkins CI) - that's just from the public pipeline.
Each retry extends the build time and increases the size of the build queue for macOS/iOS nodes, causing everyone on the dev/qa teams to wait longer.
Those tests need to consistently pass without a retry for the entire testing stage.
PS. We just added the same retry for network-audit on macos x64, for the same reason.
Description
Browser tests have their own built-in retry mechanism, but this is not sufficient on its own. We had to rebuild often enough, that we ended up adding an automated retry in CI for the entire browser test stage, including a notification whenever it's triggered. In the last 3 months we got ~160 such notifications (Slack search:
"test-browser" "was retried" "times on" in:#devops-bot-dev from:@Jenkins CI
) - that's just from the public pipeline.Each retry extends the build time and increases the size of the build queue for macOS/iOS nodes, causing everyone on the dev/qa teams to wait longer.
Those tests need to consistently pass without a retry for the entire testing stage.
PS. We just added the same retry for network-audit on macos x64, for the same reason.