Closed MadhavJivrajani closed 5 months ago
Thank you very much for opening this issue.
Just slightly more context. In Cluster API we had flaky jobs for the last 1-2 years, it was just at a rate of ~ <5% so we didn't push the GitHub token issue with the highest priority (so I assume more IPs alone wouldn't help, as you wrote).
If I'm connecting the dots correctly we would provide the token to the ProwJobs via ExternalSecrets. As far as I'm aware ExternalSecrets are not yet available on the EKS clusters (but there is or will be a discussion about that, cc @ameukam)
If I'm connecting the dots correctly we would provide the token to the ProwJobs via ExternalSecrets. As far as I'm aware ExternalSecrets are not yet available on the EKS clusters (but there is or will be a discussion about that, cc @ameukam)
ExternalSecrets are available in the EKS Prow build cluster. We can source secrets from the AWS Secrets Manager in the Prow AWS account and from the GCP Secrets Manager in the k8s-infra-prow-build
account.
Update: we migrated all nodes in the EKS Prow build cluster to a public subnet, so all nodes have public IP addresses instead of routing all traffic via a NAT Gateway. That should significantly improve the situation, but if you still see increased failure rate due to rate limits, please let us know.
Just adding here for record -- discussion from k8s slack channel #sig-k8s-infra around Nodes are randomly freezing and failing
- https://kubernetes.slack.com/archives/CCK68P2Q2/p1693476605123389
We should probably create a new bot. (k8s-contribex-ci-robot
? ) operated by the github admin team to provide tokens requested by the community.
I don't think we need a separate account for this. With the changes made to the eks cluster and switching to authenticated requests we realistically won't have other requests. We'll just have this one option for people to use for authenticated read-only requests.
I think we can create a new account if more requests like this come up. @ameukam can we generate a token from the k8s-infra-ci-robot for this? The governance of this token can be under the purview of github-admins if needed. Thoughts?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
Context
There has been an uptick in projects seeing flakiness in their tests running in either the GKE or EKS clusters due to the tests hitting GitHub rate limits.
Issues and discussions around this:
It seems like these jobs pull artifacts/files needed using the GitHub APIs. Its worth noting that cloning repositories itself does not count against the rate limit: https://github.com/orgs/community/discussions/44515.
It has also been reported that this issue of rate limiting has been exacerbated by moving jobs to the EKS cluster: https://github.com/kubernetes/org/issues/4165#issuecomment-1676985033. This isn't surprising since nodes on EKS clusters have private IPs and the traffic is egress-ing through NAT and Internet gateways because of which the number of IPs hitting GitHub is very low. There is discussion in SIG K8s Infra around assigning public IPs to nodes, similar to how GKE does it: https://github.com/kubernetes/k8s.io/issues/5759, this would not only help GitHub rate limits, but also rate limits observed while pulling from Docker registry.
However, moving to public IPs also might not prove sufficient for the GitHub rate limit issue for pull heavy jobs since some of them have been experiencing this issue even on the GKE cluster (https://github.com/kubernetes/org/issues/4165).
It has also been suggested to use
ghproxy
to get around this GitHub rate limit issue: https://github.com/kubernetes/org/issues/4165#issuecomment-1525504515. However, the issue with this is implementations of non-prow clients might have to be significantly changed to adapt toghproxy
(https://github.com/kubernetes/org/issues/4165#issuecomment-1525516684).Proposal
This issue hopes to track the discussion and decision around creating a new read only GitHub token that can be shared (similar to how some jobs re-use bot tokens) by projects with read heavy jobs against the GitHub API.
Authenticated requests have a rate limit of 5000 requests/hour/account, which should be a sufficient aggregate limit.
Prior art: https://github.com/kubernetes/k8s.io/pull/4259
/sig k8s-infra testing contributor-experience /area github-management /cc @ameukam @xmudrii @kubernetes/owners