search

conda-forge / conda-forge.github.io

The conda-forge website.
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
120 stars 271 forks source link

github api rate exceeded #908

Open casperdcl opened 4 years ago

casperdcl commented 4 years ago

Issue:

conda-forge making excessive GitHub API calls (hitting the rate limit of 5,000/hr) using all conda-forge user's tokens via "Travis CI for Open Source."

It's unclear if this is an issue with Travis or with conda-forge or both. It's killing all my automation which depends on being able to access the GitHub API.


Details about issue:

> Hello MK, > > The original response from Travis CI was that the conda-forge org was only responsible for about 100 API calls. > > The conda-forge org has a massive number of github users (I'm sure you know what conda-forge is) so I'm skeptical that this is an issue at their end - it would mean thousands of people should be experiencing the same issue as me. > > Personally I only own a couple of conda-forge repos so don't see how the thousands of other repos it owns should affect me or use my tokens for builds. > > In any case I've opened an issue here: https://github.com/conda-forge/conda-forge.github.io/issues/908 > > Regards, > Casper -- @casperdcl, Oct 24, 00:40 UTC > Hello Casper, > > Thanks for writing in and sorry for the inconvenience this has caused. > > It appears your token may be in use across the conda-forge organization. We also observed that you have builds running for conda-forge on .ORG and .COM. As a result, all requests for "Travis CI for Open Source" are being triggered with this token, can you confirm this? In addition, please can you have Github send you the list of repositories using this token? -- MK (Travis CI), Oct 23, 12:05 EDT > Hey Casper, > > Thanks for following and for sending these additional details about your issue. > > I confirm we are seeing a lot of activity for your account on Sunday in our logs. > > I'll raise this to our Engineering Team to see what they think. > > Thank you for your patience. -- Dominic Jodoin (Travis CI), Oct 3, 16:50 EDT > Dear Mustafa, > > Unfortunately GitHub Develper support was very explicit that this issue is caused by the OAuth app "Travis CI for Opens Source" by travis-ci (https://github.com/settings/connections/applications/f244293c729d5066cf27 from travis-ci.org). To be clear, this is NOT an issue with the OAuth app "Travis CI" by travis-pro (https://github.com/settings/connections/applications/88c5b97de2dbfc50f3ac from travis-ci.com), nor the GitHub app "Travis CI" by travis-ci (https://github.com/settings/installations/1269142 from ???). > > I do not have control over the API requests that "Travis CI for Open Source" makes. To be clear, it would be helpful if you could: > > - let me know if there's any repository/issue tracker for this sort of problem on https://github.com/travis-ci or similar > - let me know what the differences are between the 3 different Travis apps mentioned above > - forward this to a developer of"Travis CI for Open Souce" to check for bugs (e.g. infinite loop making API requests) > - let me know how many GitHub payloads/requests my account sends to Travis per hour which may trigger Travis to request data via the GitHub API (I'm sure this is a small number which should be nowhere near 5,000/hour) > > I'm sorry but this is very urgent, core to Travis CI services, and something which may affect every single one of your customers. Please let me know if this is not the case. I maintain some of the world's most popular open source repositories and will be very vocally switching to > alternative providers if this issue cannot be rectified. > > Best, > Casper -- @casperdcl, Oct 3, 09:00 EDT
isuruf commented 4 years ago

@casperdcl, this is a known issue with Travis-CI. There's nothing we can do. We can't migrate to .COM because ppc64le is only available on .ORG.

casperdcl commented 4 years ago

@isuruf thanks a lot for you repsonse. is this:

  1. purely an issue with @travis-ci (i.e. even if I had nothing to do with @conda-forge there'd still be 5k/hr calls), or
  2. an issue with how @travis-ci handles orgs such as @conda-forge (i.e. lots of members and repos etc.), or
  3. an issue with how @conda-forge uses @travis-ci to do what it considers essential?
isuruf commented 4 years ago

It's 2. Note that this happens to all members of conda-forge. If you weren't a member of @conda-forge, but you were a member of 2 orgs each with 3K repos, you'd still have this issue.

Travis-CI synchronizes the permissions of each user every day I think and for each repo in each org that the user is a part of, they try to get the permission that you have and if you get write access for a repo, then your permissions in travis-ci is also updated. This is 1 API call per repo and therefore 8K calls are made.

btovar commented 4 years ago

Our automation is also being hit by this. Is there a known workaround?

scopatz commented 4 years ago

you can buy a higher api limit

jakirkham commented 4 years ago

Really? How do you do that?

casperdcl commented 4 years ago

I feel like a possible work-around would be if GitHub could increase the API rate for @conda-forge members... It's a little ridiculous that open source devs are being penalised for being open source devs. I'm seriously considering leaving the @conda-forge org to avoid this issue.

isuruf commented 4 years ago

This is probably fixed for non-@conda-forge/core members. I removed read access for all repositories and you should have permissions only for the repositories you maintain. Your first Travis-CI sync will still exhaust your github api rate, but subsequent ones shouldn't. (That's what I think should happen if my guess about how Travis-CI uses the token is correct)

casperdcl commented 4 years ago

Ok, though that still sounds like it would be impossible to complete the initial sync then. After 5k is exhausted, will it resume the sync for the remaining 3k 1 hour later?

isuruf commented 4 years ago

I think you'll have to manually start the sync 1 hour later (Or wait 24 hours and Travis-CI will do it for you)

isuruf commented 4 years ago

I think Travis-CI persists the state after each call, otherwise we wouldn't get permission earlier on travis-ci. So, a re-run should work.

casperdcl commented 4 years ago

@isuruf I hope this issue can be closed then. Was incredibly hard to debug and involved multiple support tickets with 3 different companies. I'm going to lie down now :)

isuruf commented 4 years ago

Let's keep this open until we can confirm that this works. If you don't see any repositories you don't maintain in https://travis-ci.org/organizations/conda-forge/repositories, then we can close this. I can't test because I have write access to all 8k+ repos.

casperdcl commented 4 years ago

hmm says "Last synced about a minute ago" when I hover over the "Sync account" button, and still has 339 pages under "Legacy Services Integration" for conda-forge. Afraid to click "Sync account."

isuruf commented 4 years ago

@casperdcl, did this get fixed?

casperdcl commented 4 years ago

I still have 342 pages on https://travis-ci.org/organizations/conda-forge/repositories but haven't encountered the rate limit error recently.