fastai / ghapi

A delightful and complete interface to GitHub's amazing API
https://ghapi.fast.ai/
Apache License 2.0
527 stars 57 forks source link

ghapi.page.paged does not respect rate limits #110

Open Ark-kun opened 2 years ago

Ark-kun commented 2 years ago

Calling it almost always results in HTTP403ForbiddenError: HTTP Error 403: Forbidden

for result in ghapi.page.paged(gh_api.search.code, q=f"filename:component.yaml repo:Ark-kun/pipeline_components"):
    print(len(result["items"]))
opt/conda/lib/python3.7/site-packages/ghapi/core.py in __call__(self, path, verb, headers, route, query, data)
    107             for k,v in route.items(): route[k] = quote(str(route[k]))
    108         res,self.recv_hdrs = urlsend(path, verb, headers=headers or None, debug=self.debug, return_headers=True,
--> 109                                      route=route or None, query=query or None, data=data or None)
    110         if 'X-RateLimit-Remaining' in self.recv_hdrs:
    111             newlim = self.recv_hdrs['X-RateLimit-Remaining']

/opt/conda/lib/python3.7/site-packages/fastcore/net.py in urlsend(url, verb, headers, route, query, data, json_data, return_json, return_headers, debug)
    210         return urlread(req, decode=False, return_json=False, return_headers=return_headers)
    211 
--> 212     return urlread(req, return_json=return_json, return_headers=return_headers)
    213 
    214 # Cell

/opt/conda/lib/python3.7/site-packages/fastcore/net.py in urlread(url, data, headers, decode, return_json, return_headers, timeout, **kwargs)
    111         with urlopen(url, data=data, headers=headers, timeout=timeout, **kwargs) as u: res,hdrs = u.read(),u.headers
    112     except HTTPError as e:
--> 113         if 400 <= e.code < 500: raise ExceptionsHTTP[e.code](e.url, e.hdrs, e.fp) from None
    114         else: raise
    115 

HTTP403ForbiddenError: HTTP Error 403: Forbidden
akvadrako commented 1 year ago

I also am getting lots of secondary rate limits. It would be great if at least I could access the Retry-After header that github sets. Otherwise I just need to guess.