Closed Renari closed 4 years ago
I think this may turn out to be normal case since pixiv limits its api rate.
I am having same issue sometimes, I try to solve this by adding retry mechanisms and limit api rate.
BTW, it seems yesterday pixiv had a network congestion, not sure it related to this issue.
Hm, strange since it's failing for all requests and none of them fail locally.
Will you provide a testing code? It still may help even it may not re-produces the error you described.
Also please check your account is valid (not using 3rd o-auth), because I used to login with my google account, but some how one day it stop working. Then I realize that I wasn't suppose to use google o-auth to login, and the reason why it works in past become a mystery.
My code can be found here: https://github.com/Renari/saber-alter/blob/master/src/index.ts#L87
I'll see if I can make a more self contained example.
Here's a simpler example: https://github.com/Renari/pixiv-grabber/blob/master/index.js
This runs fine locally but on my server it stalls for like 30 seconds before timing out with an error.
The simpler example works fine on my server. Perhaps you can try on different server? If that works, perhaps your ip or server has been banned (However I never heard this case)
It's looking like some sort of ban is likely since I can't access their domain at all:
traceroute to pixiv.net (210.140.131.225), 30 hops max, 60 byte packets
1 * * *
2 209.222.14.193 (209.222.14.193) 1.321 ms 3.159 ms 3.154 ms
3 * * *
4 * * *
5 ae-33.a00.nycmny17.us.bb.gin.ntt.net (128.241.2.201) 12.827 ms ae-35.a01.nycmny17.us.bb.gin.ntt.net (128.241.2.249) 1.486 ms ae-33.a00.nycmny17.us.bb.gin.ntt.net (128.241.2.201) 12.771 ms
6 * * *
7 4.7.26.70 (4.7.26.70) 61.976 ms * *
8 ae-28.r05.sttlwa01.us.bb.gin.ntt.net (129.250.2.45) 59.315 ms 59.253 ms 203.181.106.177 (203.181.106.177) 61.976 ms
9 ae-0.a01.sttlwa01.us.bb.gin.ntt.net (129.250.5.86) 60.803 ms 69.188 ms ae-1.a01.sttlwa01.us.bb.gin.ntt.net (129.250.5.98) 63.344 ms
10 ae-0.iij.sttlwa01.us.bb.gin.ntt.net (131.103.116.22) 59.146 ms 59.068 ms 27.86.46.93 (27.86.46.93) 181.189 ms
11 tky001bb10.IIJ.Net (58.138.88.129) 155.319 ms 155.906 ms oteJIN301.int-gw.kddi.ne.jp (27.86.32.2) 174.751 ms
12 tky001ix51.IIJ.Net (58.138.102.26) 155.777 ms 155.757 ms tky001ix51.IIJ.Net (58.138.102.30) 180.895 ms
13 210.130.154.166 (210.130.154.166) 195.020 ms 166.923 ms 193.157 ms
14 202.93.95.182 (202.93.95.182) 150.010 ms 151.466 ms 118.151.224.194 (118.151.224.194) 165.362 ms
15 118.151.224.194 (118.151.224.194) 152.914 ms 158.205.134.62 (158.205.134.62) 183.381 ms 118.151.224.194 (118.151.224.194) 183.099 ms
16 158.205.134.62 (158.205.134.62) 195.112 ms 193.543 ms *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
So I've tried a few different servers now from vultr, seems all of their servers are blacklisted by pixiv.
Tried one from digitalocean seems they're blacklisted as well.
Perhaps you can try upbit/pixivpy for double checking?
>>> from pixivpy3 import *
>>> api = AppPixivAPI()
>>> json_result = api.illust_detail(59580629)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/aapi.py", line 146, in illust_detail
r = self.no_auth_requests_call('GET', url, params=params, req_auth=req_auth)
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/aapi.py", line 40, in no_auth_requests_call
self.require_auth()
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 57, in require_auth
raise PixivError('Authentication required! Call login() or set_auth() first!')
pixivpy3.utils.PixivError: Authentication required! Call login() or set_auth() first!
>>> api.login(USERNAME, PASSWORD)
Traceback (most recent call last):
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connection.py", line 334, in connect
conn = self._new_conn()
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/saberalter/.local/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/saberalter/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 67, in requests_call
**self.requests_kwargs)
File "/home/saberalter/.local/lib/python3.5/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/saberalter/.local/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/saberalter/.local/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/saberalter/.local/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 81, in login
return self.auth(username=username, password=password)
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 117, in auth
r = self.requests_call('POST', url, headers=headers, data=data)
File "/home/saberalter/.local/lib/python3.5/site-packages/pixivpy3/api.py", line 72, in requests_call
raise PixivError('requests %s %s error: %s' % (method, url, e))
pixivpy3.utils.PixivError: requests POST https://oauth.secure.pixiv.net/auth/token error: HTTPSConnectionPool(host='oauth.secure.pixiv.net', port=443): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa4c44df8d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
You should login before any api call. I believe the example on the readme needs update.
from pixivpy3 import *
api = AppPixivAPI()
api.login('?????', '?????')
# get origin url
json_result = api.user_illusts(4752417)
print(json_result)```
Output:
{'illusts': [{'id': 78794602, 'title': '2019 December All Rewards packag', 'type': 'illust', 'image_urls': {'square_medium': 'https://i.pximg.net/c/540x540_10_webp/img-master/img/2020/01/09/02/29/51/78794602_p0_square1200.jpg', 'medium': 'https://i.pximg.net/c/540x540_70/img-master/img/2020/01/09/02/29/51/78794602_p0_master1200.jpg', 'large': 'https://i.pxim ........
I did login after that, the login timed out.
Well, then I guess your vps is actually blocked :(
From what I've tested most vps are blocked 😢
FYI, two weeks ago I tested on Oracle Cloud. Works fine for me.
However I lost my machine so I can't test now.
OK, I did some more testing:
vultr | digitalocean | oraclecloud | |
---|---|---|---|
traceroute | :x: | :x: | :x: |
wget | :x: | ✔️ | ✔️ |
So I believe digitalocean and oraclecloud would both work.
Although I noticed oraclecloud offers a free version which seems fine for this purpose.
node
version: 12.14.1npm
(oryarn
) version: 6.13.6Do you want to request a feature or report a bug?:
What is the current behavior?: https://gist.github.com/Renari/ade19f8b4d28a5ad19f10253a40eb7e3
The above is constantly getting output to my log when deploying my code. I'm having difficulty reproducing the issue locally though.
What is the expected behavior?: Requests to not time out.
Suggested solution: Unknown