InnerSourceCommons / InnerSourceLearningPath

Holds the source material for the InnerSource Commons Learning Path
https://innersourcecommons.org/learningpath
Creative Commons Attribution Share Alike 4.0 International
74 stars 46 forks source link

503 throttle graphql requests #512

Closed marshmallowrobot closed 2 years ago

marshmallowrobot commented 2 years ago

This PR introduces the throttling plugin to our GraphQL calls. It prevents the rate-limiting errors we were seeing prior to this, but at the cost of increased time. It now takes a couple of minutes for scripts/generate_learning_path_markdown.js to complete.

The throttling plugin requires two new functions to be defined: onRateLimit() and onSecondaryRateLimit(), neither of which should get triggered if the throttling is working as intended. I didn't do anything different from the documentation in those.

I also made small changes to the auth and query objects. The @octokit README specifically mentions string template literals as an anti-pattern, so I removed them, and in the query, used a variable instead.

rrrutledge commented 2 years ago

This work is amazing, @marshmallowrobot ❗️ I added @tsadler1988 to this pull request, who is the original author of a lot of this code. I’m excited to take a look, too!

marshmallowrobot commented 2 years ago

Great stuff! So does the build actually finish now?

Locally, YES! It does. Takes 2.5-3 min to finish, but its consistent, and no rate limit responses from GitHub API

lenucksi commented 2 years ago

Great stuff! So does the build actually finish now?

Locally, YES! It does. Takes 2.5-3 min to finish, but its consistent, and no rate limit responses from GitHub API

Sounds great, how about we try it? I still remember a 10 minute pre-commit script, (not CI, just pre-commit) so 2.5-3 minutes sounds good to me. (Not that one should compare themselves to that pre-commit script. Company was fine, effect of the script was also useful, still too long. :wink: )

marshmallowrobot commented 2 years ago

Sounds great, how about we try it?

Ok! What do I do to try it? If I merge the PR, does that magically kick something off? Or do I have to go run another script somewhere? What's this process like?

lenucksi commented 2 years ago

Sounds great, how about we try it?

Ok! What do I do to try it? If I merge the PR, does that magically kick something off? Or do I have to go run another script somewhere? What's this process like?

From what I can see here: https://github.com/InnerSourceCommons/InnerSourceLearningPath/blob/main/.github/workflows/publish-to-website.yml it should run. However, it appears to only run if certain directories have been touched by a commit. So might need a manual run.

marshmallowrobot commented 2 years ago

@lenucksi it worrrrrrked! like maaaagic! TY!

And TY! @rrrutledge for the timely PR review and approval ^_^

rrrutledge commented 2 years ago

Woah! So amazing? So the website is updated now?

rrrutledge commented 2 years ago

I see it! I see it! Wow‼️

lenucksi commented 2 years ago

@lenucksi it worrrrrrked! like maaaagic! TY!

And TY! @rrrutledge for the timely PR review and approval ^_^

Boom, magic! Amazing stuff :grinning: :rocket: