NerdWalletOSS / shepherd

A utility for applying code changes across many repositories.
Apache License 2.0
231 stars 39 forks source link

Rate limiting issues pulling from GitHub #57

Closed FoodProduct closed 6 years ago

FoodProduct commented 6 years ago

When fetching a very long list of matching repos from GitHub, a consumer will run into a rate limiting error during shepherd checkout.

dbarnes:shepherd-migrations dbarnes$ shepherd checkout migrations/2018-09-cdn-url-updates/
{ HttpError: {
  "documentation_url": "https://developer.github.com/v3/#abuse-rate-limits",
  "message": "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."
}

    at response.text.then.message (/srv/nerdwallet/shepherd/node_modules/@octokit/rest/lib/request/request.js:72:19)
    at process._tickCallback (internal/process/next_tick.js:103:7)
  name: 'HttpError',
  code: 403,
  status: undefined,
  headers:
   { 'access-control-allow-origin': '*',
     'access-control-expose-headers': 'ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval',
     connection: 'close',
     'content-encoding': 'gzip',
     'content-security-policy': 'default-src \'none\'; base-uri \'self\'; block-all-mixed-content; connect-src \'self\' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src assets-cdn.github.com; form-action \'self\' github.com gist.github.com; frame-ancestors \'none\'; frame-src render.githubusercontent.com; img-src \'self\' data: assets-cdn.github.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src \'self\'; media-src \'none\'; script-src assets-cdn.github.com; style-src \'unsafe-inline\' assets-cdn.github.com',
     'content-type': 'application/json; charset=utf-8',
     date: 'Mon, 24 Sep 2018 21:30:00 GMT',
     'expect-ct': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"',
     'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
     'retry-after': '60',
     server: 'GitHub.com',
     status: '403 Forbidden',
     'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
     'transfer-encoding': 'chunked',
     'x-content-type-options': 'nosniff',
     'x-frame-options': 'deny',
     'x-github-media-type': 'github.v3; format=json',
     'x-github-request-id': '5A8C:0D02:256FFC7:2EBD74C:5BA95758',
     'x-runtime-rack': '0.007640',
     'x-xss-protection': '1; mode=block' } }

Proposal: Try/catch when fetching and obey the retry-after header returned from GitHub before continuing to paginate through the list of results. https://developer.github.com/v3/guides/best-practices-for-integrators/

parshap commented 6 years ago

Fixed by #59.