Implement retry-backoff for github API

amarza-rh commented 5 years ago

Github API has a rate limit in place and reviewrot breaks that, leading to the following exception:

DEBUG:github.Requester:GET https://api.github.com/users/nirzari  {'User-Agent': 'PyGithub/Python'} None ==> 403 {'status': '403 Forbidden', 'x-ratelimit-remaining': '0', 'x-github-media-type': 'github.v3; format=json', 'content-security-policy': "default-src 'none'", 'x-content-type-options': 'nosniff', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type', 'transfer-encoding': 'chunked', 'x-github-request-id': 'REDACTED', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'server': 'GitHub.com', 'content-encoding': 'gzip', 'x-ratelimit-limit': '60', 'x-xss-protection': '1; mode=block', 'date': 'Tue, 02 Jul 2019 08:14:42 GMT', 'access-control-allow-origin': '*', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-type': 'application/json; charset=utf-8', 'x-frame-options': 'deny', 'x-ratelimit-reset': '1562055316'} {"message":"API rate limit exceeded for 66.187.233.202. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://developer.github.com/v3/#rate-limiting "}
Traceback (most recent call last):
  File "/usr/bin/review-rot", line 4, in <module>
    __import__('pkg_resources').run_script('review-rot==1.0', 'review-rot')
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1460, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python2.7/site-packages/review_rot-1.0-py2.7.egg/EGG-INFO/scripts/review-rot", line 286, in <module>

  File "/usr/lib/python2.7/site-packages/review_rot-1.0-py2.7.egg/EGG-INFO/scripts/review-rot", line 105, in main

  File "build/bdist.linux-x86_64/egg/reviewrot/githubstack.py", line 46, in request_reviews
  File "build/bdist.linux-x86_64/egg/github/MainClass.py", line 233, in get_user
  File "build/bdist.linux-x86_64/egg/github/Requester.py", line 275, in requestJsonAndCheck
  File "build/bdist.linux-x86_64/egg/github/Requester.py", line 286, in __check
github.GithubException.RateLimitExceededException: 403 {u'documentation_url': u'https://developer.github.com/v3/#rate-limiting' , u'message': u"API rate limit exceeded for 66.187.233.202. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"}

There are several things to do here, in order of importance:

implement retry with exponential backoff for github API
- this should work in the general case
parse the rate limit response from Github and respect that (wait for the designated period) and re-try the request
~add support for authenticated requests to Github API -- this permits a higher rate of polling Github API~

redhat-aqe / review-rot

Implement retry-backoff for github API #104