Making over a 100 concurrent requests to GitHub API

[sayanee]

I wanted to reach out to let you know about some changes we are making to the secondary (anti-abuse) rate limits for the GitHub API today, which are documented here:

https://developer.github.com/v3/guides/best-practices-for-integrators/#dealing-with-abuse-rate-limits

We noticed that OAuth Application webuild with App ID 114198 is sometimes making over a 100 concurrent requests to the GitHub API, which isn't something we can support currently. We wanted to give you a heads up that the changes we'll make to the anti-abuse rate limits will temporarily limit actors that make requests at such high concurrency (as already described in the documentation).

In order to not hit those limits, you'll need to reduce the concurrency at which you're making requests -- before making more API requests, you'll need to wait for some previous API requests to finish so that you don't have lots of pending requests open. We can't give you an exact number here since the limits are not that simple and we'll be tweaking them over time, but I'd recommend reducing concurrency as much as possible, e.g. so that you don't make over ~20 requests concurrently, and then reducing further if you notice that you're still hitting these limits.

We understand that you're not making those requests with abusive intentions (which is why we're reaching out to let you about this), but we're making these changes to help us keep GitHub fast and available for all our users and applications. Large spikes in concurrent requests make that more difficult and sometimes cause problems affecting other users.

If you have any questions feel free to let me know.

[notthetup]

There are two places when multiple Github requests are in flight at the same time.

https://github.com/webuildorg/webuild-repos/blob/master/repos/index.js#L286

and

https://github.com/webuildorg/webuild-repos/blob/master/repos/index.js#L216

... these need to be serialised to a certain extent.

Using something like https://github.com/charto/cwait could help to limit the number of calls being done in parallel.

[sayanee]

More information:

It looks like there were 634 concurrent requests made via the We Build OAuth application (at the time we made the log query, last Friday -3 days) via the user agent NodeJSHTTP Client.

[jsstrn]

If no one is working on this, I could give it a shot. ✌️😊

[notthetup]

We're also seeing this:

Error for https://github.com/isovic/graphmap: {
  "documentation_url": "https://developer.github.com/v3/#abuse-rate-limits",
  "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."
}

Error for https://github.com/stepango/Archetype: {
  "documentation_url": "https://developer.github.com/v3/#abuse-rate-limits",
  "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."
}

Error for https://github.com/Yaoshicn/decaptcha: {
  "documentation_url": "https://developer.github.com/v3/#abuse-rate-limits",
  "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."
}

Error for https://github.com/MugunthKumar/MKStoreKit: {
  "documentation_url": "https://developer.github.com/v3/#abuse-rate-limits",
  "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."
}

https://developer.github.com/v3/#abuse-rate-limits

[jsstrn]

So I'd like to write some tests so I know my implementation works, but it seems like the code isn't currently being tested. What's the best way to test this?

I was thinking I could maybe stub everything and listen to the Promise calls. Any suggestions? Or would you prefer I submit my changes (dare I say it) without tests? 🙀

[sayanee]

haha just send whatever you have in a pull request to a branch and then we can chat on the code / whether to test? 😉

Thank you @jsstrn 🤗