"rate limit exceeded" for datalad-installer --sudo ok git-annex -m datalad/git-annex:release

yarikoptic commented 1 year ago

see e.g. https://github.com/DanielDent/git-annex-remote-rclone/actions/runs/3750292044/jobs/6370226146 for https://github.com/DanielDent/git-annex-remote-rclone/pull/59

File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/datalad_installer.py", line 2386, in get_latest_release_asset
    for release in self.paginate(f"[https://api.github.com/repos/{repo}/releases](https://api.github.com/repos/%7Brepo%7D/releases)"):
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/datalad_installer.py", line 2301, in paginate
    with self.get(url) as r:
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/datalad_installer.py", line 2292, in get
    with urlopen(req) as r:
         ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error [40](https://github.com/DanielDent/git-annex-remote-rclone/actions/runs/3750292044/jobs/6370226146#step:5:41)3: rate limit exceeded
Error: Process completed with exit code 1.

apparently it is a "new" issue in this repo, and which is odd ! could we then accompany that crash with status about rate limits we have at that point like we do in cron job for tinuous on smaug

curl -fsSL -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/rate_limit

to see if we do really? or should we just retry few times?

jwodder commented 1 year ago

@yarikoptic In this case, the rate limit was likely exceeded because no GITHUB_TOKEN envvar was set, and so datalad-installer was making unauthenticated requests.

yarikoptic commented 1 year ago

please then set it there to some token for that repo -- I think you have full set of permissions there.

jwodder commented 1 year ago

@yarikoptic PR to set token: https://github.com/DanielDent/git-annex-remote-rclone/pull/60

yarikoptic commented 1 year ago

Let's address this issue by catching exception and if it talks about rate limit exceeded :

if GITHUB_TOKEN was set and provided in the query -- do query to https://api.github.com/rate_limit and provide output of it before reraising that exception
if GITHUB_TOKEN was not set and thus not provided -- issue an alternative RuntimeError exception which would inform user to set GITHUB_TOKEN to not hit such rate limits for non-authenticated access

datalad / datalad-installer

"rate limit exceeded" for datalad-installer --sudo ok git-annex -m datalad/git-annex:release #137