BackgroundSync being rate limited

[joannenolan-sky]

Describe the Bug Unable to perform backgroundSync as application is being rate limited when getting status or reviews. This happens for each open pull request

14:16:22.118Z ERROR wuffle:background-sync:
  additional sync failed RequestError [HttpError]: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.
      at /my-wuffle/packages/app/node_modules/@octokit/request/dist-node/index.js:66:23
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:85:5)
      at async Job.doExecute (/my-wuffle/packages/app/node_modules/bottleneck/light.js:405:18) {
    name: 'HttpError',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type',
      connection: 'close',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com",
      'content-type': 'application/json; charset=utf-8',
      date: 'Tue, 15 Oct 2019 14:16:22 GMT',
      'expect-ct': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      'retry-after': '60',
      server: 'GitHub.com',
      status: '403 Forbidden',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': '0857:2FD5A:3F6880E:4C0E0DA:5DA5D4B4',
      'x-xss-protection': '1; mode=block'
    },
    request: {
      method: 'GET',
      url: 'https://api.github.com/repos/my-repo/repo/pulls/766/reviews',
      headers: {
        accept: 'application/vnd.github.v3+json',
        'user-agent': 'octokit.js/16.30.1 Node.js/12.9.1 (macOS Mojave; x64)',
        authorization: 'token [REDACTED]'
      },
      request: {
        hook: [Function: bound bound register],
        validate: [Object],
        retryCount: 1
      }
    },
    documentation_url: 'https://developer.github.com/v3/#abuse-rate-limits'
  }

Steps to Reproduce

valid configuration Setup : 7 repos, 900 total issues(750 pull requests)(9 open pull requests) Version 17 storedump.json running on latest(v27) of application

• npm clean-install
• npm run buildl
• npm run start

Expected Behavior

full refresh of issues should happen, reviewers and status should be updated wit no issues

Environment

running in a local environment

• Browser: [e.g. Chrome 77.0.3865.90]
• OS: [e.g. MacOS Mojave]
• Board Version: [e.g. 0.27.0]

[joannenolan-sky]

This also happens when there is no storedump.json file present

[nikku]

Did you add the new required permissions on status, checks and so on? Please check your log for WARN entries.

[nikku]

Check this diff for the newly required permissions.

[joannenolan-sky]

Yes, I need to add the webhook permission but I already had the checks and status added . I created a new app to check and compare the permissions to make sure they matched.

I have my logs outputted to a file attached. The config is valid and there is no warning except for the abuse limits wuffle.log

[nikku]

Thanks for sharing the log, is it helpful.

[nikku]

A few insights of my preliminary analysis: GitHub rate-limits to 5000 requests per hour. Looking into your setup, we get decently close to that boundary with a single sync already.

There are a few ways to work around this limitation:

• Only sync open issues (will still fail / conflict with rate limits at some point)
• Find a clever sync strategy that adjusts the synchronization rate to never hit the rate limit

Additional idea:

• We should always prioritize open issues when synchronizing

[mjcarroll]

I've recently run into this as well in a large organization. Prioritizing open issues as well as newer issues would be a great help.

[nikku]

A few more details on how background synchronization works that relates to this issue:

Background synchronization handles open and closed issues differently: Open issues are always synchronized, closed issues are only synchronized if they have been updated in the last 30 days. This is obviously a problem in test setups or super huge organizations that have 1000 or more issues open or closed in thirty days.

https://github.com/nikku/wuffle/commit/31c274758594bdd1d9161ee5d3d28f262f1fbd58 and https://github.com/nikku/wuffle/commit/9dd0c636fbbb93da0c06117ac6350e8cd6b3280f ship some additional changes:

• Details (checks, statuses, reviewers) are only being synchronized if issues have recently been updated (in last four hours for closed issues, in the last day for open issues)
• Updating an issue on GitHub or on the board ensures that it is being picked up by background sync and synched including their details with the next sync check.
• Those things can be fine-tuned via environment variables to suite specific needs and balance open versus closed issue synchronization and initial sync behavior (cf. commits mentioned above).

As the result of this change, initial is going to fetch details for recently updated issues only. As issues are being worked on (i.e. changed via the board or GitHub) details will eventually be background synched.

I expect this to be a good change and look forward for your feedback how this behaves in your organizations. I got a few additional ideas for optimization, however I'd rather see the impact of this change in bigger setups (yours, ours) before dumping huge amounts of additional work in this matter.

Closing this issue for now, looking forward for your feedback.

[joannenolan-sky]

Thanks for that I have just updated to v0.29.0 and I have been rate limited yet which is great. I have tried it against with an original storedump.json present and with no storedump.json Thanks for getting that sorted

[nikku]

Glad to hear!