branchprotector continuously failing

kubernetes / test-infra

Test infrastructure for the Kubernetes project.

Apache License 2.0

3.85k stars 2.66k forks source link

branchprotector continuously failing #33900

Open BenTheElder opened 13 hours ago

BenTheElder commented 13 hours ago

It's timing out after 5h20m: https://prow.k8s.io/?job=ci-test-infra-branchprotector

This is causing us to not sync branch protection rules, e.g. related to https://github.com/kubernetes/test-infra/pull/33857 / https://github.com/kubernetes/test-infra/pull/33880

We have a big problem with this tooling not scaling, it uses a ton of API quota and needs to run continuously and scales with the number of repos/branches which is only growing.

/area prow /sig testing /priority important-soon

BenTheElder commented 13 hours ago

filed https://github.com/kubernetes/test-infra/pull/33901 as a stopgap. it's not great.

BenTheElder commented 12 hours ago

Aborted the current run and started a new re-run with the latest config and the 12h timeout, but we know it will be at least 5+ hours before we can tell if that worked as a stopgap ... https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-branchprotector/1864084620887199744