envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.9k stars 4.79k forks source link

All mobile CI is permafailing #36362

Open phlax opened 3 weeks ago

phlax commented 3 weeks ago

Not sure what the problem is - looks like the Engflow RBE has stopped working

All mobile CI is completely non-functional at this point

typical error:

ERROR: /root/.cache/bazel/_bazel_root/bb3efe6b13248f746491cbfc24ae2efe/external/com_google_absl/absl/base/BUILD.bazel:95:11: Compiling absl/base/internal/spinlock_wait.cc failed: (Exit 34): io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 59.999946509s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=10.1.103.68/10.1.103.68:8081]]]
com.google.devtools.build.lib.remote.common.BulkTransferException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 59.999946509s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=10.1.103.68/10.1.103.68:8081]]]

first occurrence is here https://github.com/envoyproxy/envoy/commit/9d1527efd892ca7dfd2819dd0ddac7602a14bc5e - which is clearly unrelated

phlax commented 3 weeks ago

fixed by Engflow

phlax commented 3 weeks ago

happening again/still - some jobs seemed to pass but it was short-lived