search

mntnr / name-your-contributors

Name your GitHub contributors; get commits, issues, and comments
MIT License
73 stars 20 forks source link

Daemonised #64

Closed tgetgood closed 6 years ago

tgetgood commented 6 years ago

This builds on #61, #62, and #63.

First step towards daemonisation. With this PR, name-your-contributors will run indefinitely, sleeping when it runs out of quota until the quota resets. Any other error response that contains a retry-after header will also cause the client to sleep that long and then try to continue.

This theoretically allows name-your-contributors to grab arbitrarily large sets of data politely over time. I've tested it but almost exhausting my quota and then grabbing something large. I'm still running larger tests, looking for something that takes more than 10k quota to see that it sleeps multiple times. With the recent query optimisations I'm actually having trouble finding a simple query that expensive.

At this point I'm as confident that I reasonably can be that it works. Give it a shot.

One caveat: GitHub does not like you making more than one request in parallel with the same API token. That means that if you run nyc in multiple terminals at once, they will, in concert set off the abuse alarm. They won't fail, they'll simply go to sleep every time the alarm goes off and try again later. This can lead to reduced performance. This isn't a new problem; it's just more obvious now that the script can run longterm.

tgetgood commented 6 years ago

I've rebased this onto master.

tgetgood commented 6 years ago

Though not very well...

RichardLitt commented 6 years ago

Conflicts? Edit: Nevermind

RichardLitt commented 6 years ago

Is this ready?

tgetgood commented 6 years ago

Yep it's all set. I lost a change when I rebased it but that's fixed now.

RichardLitt commented 6 years ago

:tada: This PR is included in version 3.4.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: