search

yanyongyu / githubkit

The modern, all-batteries-included GitHub SDK for Python, including rest api, graphql, webhooks, like octokit!
MIT License
158 stars 21 forks source link

Rate limiting mechanism #66

Open schelv opened 6 months ago

schelv commented 6 months ago

I decided to speed up my algorithm by sending multiple requests at the same time. It works fine until it doesn't because of the rate limits

It would be nice if rate limits could be taken into account when sending lots of asynchronous requests. Currently I am encountering secondary rate limits. The returned response has status 403 and has a 'retry-after' header, maybe it can be used to temporarily stop sending async requests? and continue after the time has passed?

There is also the primary rate limit. Could the headers mentioned be used to coordinate sending the async requests?

yanyongyu commented 6 months ago

Currently, githubkit is focusing on data validation, auth flows, and maybe api versioning and module lazy loading in the next major version. Rate limit will increase the complexity and may be considered later. At current time, githubkit has implemented the http caching in #61, rate limiting may need to be handled by user.

schelv commented 6 months ago

I've implemented rate limiting in my local branch. However the current implementation is kind of messy, and still requires some improvements.

I can improve the code and make a pull request for this feature. Is that something you would be interested in?

If so do you have a preference on how the rate limiting should be added? As a superclass? or as a new configuration option?

yanyongyu commented 6 months ago

I think this could be a configuration option and handle retries within the request method

honzajavorek commented 2 months ago

If https://github.com/yanyongyu/githubkit/pull/86 is merged, should this still be open? Maybe it's already done?

yanyongyu commented 2 months ago

If #86 is merged, should this still be open? Maybe it's already done?

86 only deal with the rate limit error from the github server. This issue also requests another feature that the github api request concurrency should be controlled. The request concurrency is also mentioned in the github docs and the octokit sdk features.