Build failing (rate limited)

[rrrutledge]

We can't publish to the website. See https://github.com/InnerSourceCommons/InnerSourceLearningPath/actions/runs/3228272591/jobs/5284148344

[rrrutledge]

@marshmallowrobot has some more detail.

[rrrutledge]

Some ideas:

• Can we cache results when fetching user info?
• Can we batch calls together (rather than one request per file)?

[marshmallowrobot]

Adding some more details...

The build fails in the get_contributors.js file with the following error messages:

(node:1722) UnhandledPromiseRejectionWarning: HttpError: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
[109](https://github.com/InnerSourceCommons/InnerSourceLearningPath/actions/runs/3228272591/jobs/5284148344#step:6:110)
    at /home/runner/work/InnerSourceLearningPath/InnerSourceLearningPath/InnerSourceLearningPath/scripts/node_modules/@octokit/request/dist-node/index.js:66:23
[110](https://github.com/InnerSourceCommons/InnerSourceLearningPath/actions/runs/3228272591/jobs/5284148344#step:6:111)
    at runMicrotasks (<anonymous>)
[111](https://github.com/InnerSourceCommons/InnerSourceLearningPath/actions/runs/3228272591/jobs/5284148344#step:6:112)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
[112](https://github.com/InnerSourceCommons/InnerSourceLearningPath/actions/runs/3228272591/jobs/5284148344#step:6:113)
    at async module.exports (/home/runner/work/InnerSourceLearningPath/InnerSourceLearningPath/InnerSourceLearningPath/scripts/get_contributors.js:13:49)
[113](https://github.com/InnerSourceCommons/InnerSourceLearningPath/actions/runs/3228272591/jobs/5284148344#step:6:114)
    at async /home/runner/work/InnerSourceLearningPath/InnerSourceLearningPath/InnerSourceLearningPath/scripts/generate_learning_path_markdown.js:65:30

For everyone's reference, here are GitHub's rate limit guidelines:

• Individual calls cannot request more than 500,000 total nodes.
• The GraphQL API rate limit is 5,000 points per hour. Note that 5,000 points per hour is not the same as 5,000 calls per hour. The minimum cost of a call to the GraphQL API is 1, representing a single request.
• With the GraphQL API, you can check the rate limit status by querying fields on the rateLimit object.
  • The limit field returns the maximum number of points the client is permitted to consume in a 60-minute window.
  • The cost field returns the point cost for the current call that counts against the rate limit.
  • The remaining field returns the number of points remaining in the current rate limit window.
  • The resetAt field returns the time at which the current rate limit window resets in UTC epoch seconds.
• Querying the rateLimit object returns a call's score, but running the call counts against the limit. To avoid this dilemma, you can calculate the score of a call before you run it. The following calculation works out to roughly the same cost that rateLimit { cost } returns.
  1. Add up the number of requests needed to fulfill each unique connection in the call. Assume every request will reach the first or last argument limits.
  2. Divide the number by 100 and round the result to get the final aggregate cost. This step normalizes large numbers.

[marshmallowrobot]

• I added rateLimit info to the getContributors.js query and learned that each call (now) costs 3.
• There are 169 asciidoc files. At 3 per file, that's not enough to trip the 5k/hour request limit.
• Node limit does not appear to be the issue either. Those are consistently around 20k per request/file, not enough to trip the 500k/request limit.
• We query contributors on files/paths that don't appear to display the info. Even accounting for these, it's still not enough to trip the 5k/hour request limit. Here are examples:
  • /introduction, /product-owner, etc
  • /introduction/de, /introduction/zh, etc
• Sometimes, randomly, the generation script succeeds without hitting any rate limit errors.

Did ya'll notice how the error message says "secondary rate limit"?!?? Because I TOTALLY DID NOT NOTICE THIS. Turns out a secondary rate limit is its own thing. Quoth the GitHubs:

• If you're making a large number of POST, PATCH, PUT, or DELETE requests for a single user or client ID, wait at least one second between each request.
• When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

Reading comprehension FTW!

[rrrutledge]

Great info, @marshmallowrobot ❗️ I wonder if there is some kind of open source module out there for calling GitHub that takes all of this stuff into account (with appropriate retries)?

[lenucksi]

Great info, @marshmallowrobot exclamation I wonder if there is some kind of open source module out there for calling GitHub that takes all of this stuff into account (with appropriate retries)?

I wonder if you've already tried this SDK GitHub apparently provides: https://octokit.github.io/rest.js/v19#throttling (I'd guess you're already using it, so ignore in that case. If not, maybe it's still useful.) The docs state that there appears to be a plugin to handle / facilitate handling the ratelimiter.

And there's this unrelated JS thing: https://www.npmjs.com/package/express-rate-limit and this article that does sound sort-of useful: https://www.useanvil.com/blog/engineering/throttling-and-consuming-apis-with-429-rate-limits/

[rrrutledge]

@marshmallowrobot says that the throttling module appears to work ‼️ Just need to play around with the throttling settings.

[marshmallowrobot]

@lenucksi Thank you so much for that reference! That plugin ended up being the solution!

Please check out the PR here, and let me know your feedbacks. TY! 503 throttle graphql requests

[marshmallowrobot]

If the added build time (due to throttling) is too much to bear, I have an alternative idea:

Use git command-line to fetch commiters. I know that attributing folks that don't git-commit changes was something important to ya'll. But, if we require that all contributors use GitHub going forward, then this may not be as important tomorrow as it is today.

The git shortlog command can quickly spit out a list of names and email addresses of all committers to a given file path. We could make a one-time API request for all repo users, and match the avatars up with git shortlog output to populate the contributors per file.

That said, I'm not sure how to make git commands execute from within node, or whether those commands will also be rate limited by GitHub.

[rrrutledge]

Thanks for this work and for thinking things through, @marshmallowrobot ❗️ How long is the build? I’m inclined to leave things just the way they are and wait and see if it becomes a problem.

[marshmallowrobot]

Nah it's not bad. Its like 2-3 minutes currently.

[rrrutledge]

Seems fine. A little slow for development, but we could remedy that by having an English-only build or something if needed. Seems fine for now, though.