search

brave / brave-browser

Brave browser for Android, iOS, Linux, macOS, Windows.
https://brave.com
Mozilla Public License 2.0
17.51k stars 2.27k forks source link

macos x64 browser tests should consistently pass in CI without retrying the entire testing stage #38988

Open wknapik opened 2 months ago

wknapik commented 2 months ago

Description

Browser tests have their own built-in retry mechanism, but this is not sufficient on its own. We had to rebuild often enough, that we ended up adding an automated retry in CI for the entire browser test stage, including a notification whenever it's triggered. In the last 3 months we got ~160 such notifications (Slack search: "test-browser" "was retried" "times on" in:#devops-bot-dev from:@Jenkins CI) - that's just from the public pipeline.

Each retry extends the build time and increases the size of the build queue for macOS/iOS nodes, causing everyone on the dev/qa teams to wait longer.

Those tests need to consistently pass without a retry for the entire testing stage.

PS. We just added the same retry for network-audit on macos x64, for the same reason.

wknapik commented 2 months ago

Current count: