search

akameco / pixiv-app-api

Promise base pixiv API client
https://www.npmjs.com/package/pixiv-app-api
MIT License
217 stars 29 forks source link

Requests timing out on server #30

Closed Renari closed 4 years ago

Renari commented 4 years ago

Do you want to request a feature or report a bug?:

What is the current behavior?: https://gist.github.com/Renari/ade19f8b4d28a5ad19f10253a40eb7e3

The above is constantly getting output to my log when deploying my code. I'm having difficulty reproducing the issue locally though.

What is the expected behavior?: Requests to not time out.

Suggested solution: Unknown

yanagiragi commented 4 years ago

I think this may turn out to be normal case since pixiv limits its api rate.

I am having same issue sometimes, I try to solve this by adding retry mechanisms and limit api rate.

BTW, it seems yesterday pixiv had a network congestion, not sure it related to this issue.

Renari commented 4 years ago

Hm, strange since it's failing for all requests and none of them fail locally.

yanagiragi commented 4 years ago

Will you provide a testing code? It still may help even it may not re-produces the error you described.

Also please check your account is valid (not using 3rd o-auth), because I used to login with my google account, but some how one day it stop working. Then I realize that I wasn't suppose to use google o-auth to login, and the reason why it works in past become a mystery.

Renari commented 4 years ago

My code can be found here: https://github.com/Renari/saber-alter/blob/master/src/index.ts#L87

I'll see if I can make a more self contained example.

Renari commented 4 years ago

Here's a simpler example: https://github.com/Renari/pixiv-grabber/blob/master/index.js

This runs fine locally but on my server it stalls for like 30 seconds before timing out with an error.

yanagiragi commented 4 years ago

The simpler example works fine on my server. Perhaps you can try on different server? If that works, perhaps your ip or server has been banned (However I never heard this case)

Renari commented 4 years ago

It's looking like some sort of ban is likely since I can't access their domain at all:

traceroute to pixiv.net (210.140.131.225), 30 hops max, 60 byte packets
 1  * * *
 2  209.222.14.193 (209.222.14.193)  1.321 ms  3.159 ms  3.154 ms
 3  * * *
 4  * * *
 5  ae-33.a00.nycmny17.us.bb.gin.ntt.net (128.241.2.201)  12.827 ms ae-35.a01.nycmny17.us.bb.gin.ntt.net (128.241.2.249)  1.486 ms ae-33.a00.nycmny17.us.bb.gin.ntt.net (128.241.2.201)  12.771 ms
 6  * * *
 7  4.7.26.70 (4.7.26.70)  61.976 ms * *
 8  ae-28.r05.sttlwa01.us.bb.gin.ntt.net (129.250.2.45)  59.315 ms  59.253 ms 203.181.106.177 (203.181.106.177)  61.976 ms
 9  ae-0.a01.sttlwa01.us.bb.gin.ntt.net (129.250.5.86)  60.803 ms  69.188 ms ae-1.a01.sttlwa01.us.bb.gin.ntt.net (129.250.5.98)  63.344 ms
10  ae-0.iij.sttlwa01.us.bb.gin.ntt.net (131.103.116.22)  59.146 ms  59.068 ms 27.86.46.93 (27.86.46.93)  181.189 ms
11  tky001bb10.IIJ.Net (58.138.88.129)  155.319 ms  155.906 ms oteJIN301.int-gw.kddi.ne.jp (27.86.32.2)  174.751 ms
12  tky001ix51.IIJ.Net (58.138.102.26)  155.777 ms  155.757 ms tky001ix51.IIJ.Net (58.138.102.30)  180.895 ms
13  210.130.154.166 (210.130.154.166)  195.020 ms  166.923 ms  193.157 ms
14  202.93.95.182 (202.93.95.182)  150.010 ms  151.466 ms 118.151.224.194 (118.151.224.194)  165.362 ms
15  118.151.224.194 (118.151.224.194)  152.914 ms 158.205.134.62 (158.205.134.62)  183.381 ms 118.151.224.194 (118.151.224.194)  183.099 ms
16  158.205.134.62 (158.205.134.62)  195.112 ms  193.543 ms *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
Renari commented 4 years ago

So I've tried a few different servers now from vultr, seems all of their servers are blacklisted by pixiv.

Tried one from digitalocean seems they're blacklisted as well.

yanagiragi commented 4 years ago

Perhaps you can try upbit/pixivpy for double checking?

Renari commented 4 years ago 
>>> from pixivpy3 import *
>>> api = AppPixivAPI()
>>> json_result = api.illust_detail(59580629)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/aapi.py", line 146, in illust_detail
    r = self.no_auth_requests_call('GET', url, params=params, req_auth=req_auth)
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/aapi.py", line 40, in no_auth_requests_call
    self.require_auth()
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 57, in require_auth
    raise PixivError('Authentication required! Call login() or set_auth() first!')
pixivpy3.utils.PixivError: Authentication required! Call login() or set_auth() first!
>>> api.login(USERNAME, PASSWORD)
Traceback (most recent call last):
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/saberalter/.local/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 67, in requests_call
    **self.requests_kwargs)
  File "/home/saberalter/.local/lib/python3.5/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/saberalter/.local/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/saberalter/.local/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/saberalter/.local/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 81, in login
    return self.auth(username=username, password=password)
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 117, in auth
    r = self.requests_call('POST', url, headers=headers, data=data)
  File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 72, in requests_call
    raise PixivError('requests %s %s error: %s' % (method, url, e))
pixivpy3.utils.PixivError: requests POST https://oauth.secure.pixiv.net/auth/token error: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
yanagiragi commented 4 years ago

You should login before any api call. I believe the example on the readme needs update.


from pixivpy3 import *
api = AppPixivAPI()
api.login('?????', '?????')

# get origin url
json_result = api.user_illusts(4752417)
print(json_result)```

Output: 

{'illusts': [{'id': 78794602, 'title': '2019 December All Rewards packag', 'type': 'illust', 'image_urls': {'square_medium': 'https://i.pximg.net/c/540x540_10_webp/img-master/img/2020/01/09/02/29/51/78794602_p0_square1200.jpg', 'medium': 'https://i.pximg.net/c/540x540_70/img-master/img/2020/01/09/02/29/51/78794602_p0_master1200.jpg', 'large': 'https://i.pxim ........
Renari commented 4 years ago

I did login after that, the login timed out.

yanagiragi commented 4 years ago

Well, then I guess your vps is actually blocked :(

Renari commented 4 years ago

From what I've tested most vps are blocked 😢

yanagiragi commented 4 years ago

FYI, two weeks ago I tested on Oracle Cloud. Works fine for me.

However I lost my machine so I can't test now.

Renari commented 4 years ago

OK, I did some more testing:

vultr digitalocean oraclecloud
traceroute :x: :x: :x:
wget :x: ✔️ ✔️

So I believe digitalocean and oraclecloud would both work.

Although I noticed oraclecloud offers a free version which seems fine for this purpose.