hentai-chan / hentai

Implements a wrapper class around nhentai's RESTful API.
https://www.hentai-chan.dev/projects/hentai
GNU General Public License v3.0
212 stars 20 forks source link

Library not working due to poor request #159

Open JengRung opened 2 years ago

JengRung commented 2 years ago

Describe the Bug

Getting the 'too many 503 error responses' error when calling the package

Steps To Reproduce

from hentai import Hentai

doujin = Hentai(177013)
print(Hentai.exists(doujin.id))

System Information

OS Name Python Version App Version
Windows 3.10.2 3.2.10

Error message & warnings

Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\requests\adapters.py", line 440, in send
    resp = conn.urlopen(
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
    return self.urlopen(
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
    return self.urlopen(
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 876, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "C:\Python310\lib\site-packages\urllib3\connectionpool.py", line 866, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "C:\Python310\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nhentai.net', port=443): Max retries exceeded with url: /api/gallery/177013 (Caused by ResponseError('too many 503 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\xxx\Desktop\hentai scrping\hentai\test.py", line 3, in <module>
    doujin = Hentai(177013)
  File "C:\Python310\lib\site-packages\hentai\hentai.py", line 531, in __init__
    self.__response = self.handler.get(self.api)
  File "C:\Python310\lib\site-packages\hentai\hentai.py", line 479, in get
    response = self.session.get(url, timeout=self.timeout, proxies=self.proxies or getproxies(), **kwargs)
  File "C:\Python310\lib\site-packages\requests\sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "C:\Python310\lib\site-packages\requests\sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python310\lib\site-packages\requests\sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python310\lib\site-packages\requests\adapters.py", line 510, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='nhentai.net', port=443): Max retries exceeded with url: /api/gallery/177013 (Caused by ResponseError('too many 503 error responses'))

Additional context

I know that we all have a hard time scraping nhentai due to Cloudflare. Just wondering if there is a good solution now? Also is there any instruction on how to set up OAuth2 with this package?

hentai-chan commented 2 years ago

This is a duplicate of issue #155. A developer at nhentai.net needs to implement OAuth2 first (which is something I have no control over of) so that clients (such as this hentai library) can pick up on that change and follow suit. I haven't been able to contact the site admin myself. I think they added a rule which automatically enables cloudflare protection as soon as a certain threshold is crossed which is why sometimes the API works again. Maybe if more developers try to contact them via twitter (https://twitter.com/nhentaiOfficial) a solution can be found that works for everyone.

hentai-chan commented 2 years ago

Update: There's a potential fix in #163 on its way, let's see how this plays out.

hentai-chan commented 1 year ago

Relevant comment about this issue:

https://github.com/hentai-chan/hentai/pull/163#issuecomment-1368091413