search

bazelbuild / continuous-integration

Bazel's Continuous Integration Setup
https://buildkite.com
Apache License 2.0
259 stars 136 forks source link

Flaky tests should be disabled on BazelCI #1631

Closed joeleba closed 1 year ago

joeleba commented 1 year ago
Test Example flake
src/test/java/com/google/devtools/build/lib/skyframe/LocalDiffAwarenessIntegrationTest buildkite
src/test/shell/bazel/bazel_sandboxing_networking_test buildkite
Windows tests buildkite
src/test/java/com/google/devtools/build/lib/remote/remote buildkite

IMHO these tests add quite a bit of inertia to submitting changes, especially when we have a chain of small CLs to submit.

From my own experience: the 3 automatic trials almost never resolves the flakes. It's only when we manually click the retry button from BuildKite does the result differ.

sgowroji commented 1 year ago

Adding few more flaky tests from downstream CI.

Test Example flake
src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_head_test buildkite
src/test/shell/bazel:bazel_hermetic_sandboxing_test buildkite
src/test/shell/bazel:bazel_proto_library_test buildkite
src/test/shell/bazel:bazel_workspaces_test buildkite
meteorcloudy commented 1 year ago

https://buildkite.com/organizations/bazel/analytics has just been available, those tests will probably show up on the dashboard.

meteorcloudy commented 1 year ago

I'm closing this one since all flaky tests are tracked by https://github.com/bazelbuild/bazel/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky+test%22 and gTech team is helping reporting new flaky tests based on https://buildkite.com/organizations/bazel/analytics .