Github API rate-limit hit while updating boilerplates

[jakhog]

This might just be an issue for me now as I'm experimenting with the Github APIs myself, but when I tried to update the boilerplates, it failed with this message error:

Dolittle CLI v2.0.0

info: Updating all boiler plates
info: Update boiler plate in '/Users/jakob/.dolittle/boiler-plates/ApplicationTemplate'
info: Update boiler plate in '/Users/jakob/.dolittle/boiler-plates/ArtifactTemplates'
info: Update boiler plate in '/Users/jakob/.dolittle/boiler-plates/CSharp.BoundedContext'
info: Update boiler plate in '/Users/jakob/.dolittle/boiler-plates/Veracity.CSharp.BoundedContext'
(node:93300) UnhandledPromiseRejectionWarning: TypeError: result.forEach is not a function
    at /usr/local/lib/node_modules/@dolittle/cli/bin/boilerPlates/BoilerPlatesManager.js:232:48
    at process.internalTickCallback (internal/process/next_tick.js:77:7)
(node:93300) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:93300) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I'm pretty sure that this was caused by the Github API rate-limit for unauthenticated requests. For now, I guess this isn't really a big deal, but since it's tracked by IP, it's not unthinkable that it might happen for other people in offices as well.

There are many ways to solve this, but I think for now a good start would be to check for this error, and handle it with a nice message in the terminal letting me know that I have to wait a little bit, and try again. The message you get back is well described here. You can also just try to refresh https://api.github.com/orgs/dolittle-boilerplates/repos a bunch of times yourself.

[woksin]

I've had that error before as well, it seems to be on github's side since it resolved itself after a while. But yeah, we should handle this in a nicer way

[jakhog]

Yep, that's the point of it. It actually tells you all along how many requests you can make, and when you will get more in the HTTP response headers:

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 56
X-RateLimit-Reset: 1372700873

I don't think we want to make a Github App that the users have to authorize just for this, there are three things (of top of my head) we can do:

1. Use the Github Search API, e.g. https://api.github.com/search/repositories?q=+org:dolittle-boilerplates. This has 10 times the number of requests before we hit the limit, compared to the normal APIs.
2. Cache the results (maybe just the raw JSON as a file) of the return, and only check that every now and then, and add a --force-something flag
3. Hard code a list of all the repositories with boilerplates in the CLI tool. Then this problem would go away completely, and it shouldn't really be that hard to maintain (although a bit boring).

[jakhog]

This issue should probably be posted at dolittle-tools/common. Sorry about that 🤦‍♂️

[einari]

I guess this goes away with the implementation of https://github.com/dolittle-tools/common/issues/6 - close?

[jakhog]

Yep, this is fixed now :)