y-young / nazurin

🎉 Images collection done right. Telegram 图片收藏机器人 / A Telegram bot that helps you collect ACG illustrations.
http://nazurin.readthedocs.io
MIT License
265 stars 59 forks source link

Pixiv request encountered Cloudflare Captcha #21

Closed Misaka13514 closed 3 years ago

Misaka13514 commented 3 years ago

Describe the bug

Pixiv request encountered Cloudflare Captcha on my instance deployed on Heroku.

The error given is error: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.

I did not see the relevant description after checking the documention. Should I directly report it to the upstream dependencies?

To Reproduce

Steps to reproduce the behavior:

  1. Send Pixiv link to my bot.
  2. See error

Expected behavior

works as before

Error output / Logs

Logs are usually critical to identify the cause of a bug, please paste the complete log from start to end. Make sure to remove or mask all secrets, such as bot token and personal information!

heroku logs

2021-06-08T11:58:51.059296+00:00 app[web.1]: 2021-06-08 11:58:51,059 - nazurin - INFO - Collection update: site=Pixiv, match=('90212133',)
2021-06-08T11:58:51.069183+00:00 heroku[router]: at=info method=POST path="/TELEGRAM_BOT_TOKEN" host=misakapicbot.herokuapp.com request_id=31c8bbdc-6567-4efb-9fe1-b1c95f0541fa fwd="91.108.6.58" dyno=web.1 connect=1ms service=14ms status=200 bytes=177 protocol=https
2021-06-08T11:58:51.071453+00:00 app[web.1]: 2021-06-08 11:58:51,071 - aiohttp.access - INFO - 10.97.186.122 [08/Jun/2021:11:58:51 +0000] "POST /TELEGRAM_BOT_TOKEN HTTP/1.1" 200 177 "-" "-"
2021-06-08T11:58:51.227486+00:00 app[web.1]: 2021-06-08 11:58:51,226 - nazurin - ERROR - Update {"update_id": 803740216, "message": {"message_id": 6640, "from": [private], "chat": [private], "date": 1623153530, "forward_from": [private], "forward_date": 1623126077, "text": "https://www.pixiv.net/artworks/90212133", "entities": [{"type": "url", "offset": 0, "length": 39}]}} caused <class 'pixivpy3.utils.PixivError'>: requests POST https://oauth.secure.pixiv.net/auth/token error: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.
2021-06-08T11:58:51.227860+00:00 app[web.1]: cloudscraper.exceptions.CloudflareCaptchaProvider: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.
2021-06-08T11:58:51.227866+00:00 app[web.1]: 
2021-06-08T11:58:51.227867+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2021-06-08T11:58:51.227867+00:00 app[web.1]: 
2021-06-08T11:58:51.227869+00:00 app[web.1]: pixivpy3.utils.PixivError: requests POST https://oauth.secure.pixiv.net/auth/token error: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.

Environment (please complete the following information):

deploy to heroku with default config

Additional context

I encountered this error in the last two days.

y-young commented 3 years ago

Unfortunately I can't reproduce this bug. Does this occurs constantly or occasionally? Do other Pixiv links work? Please provide your Python version on Heroku and Heroku stack version. You can also try to upgrade Python, dependencies and Heroku stack.

Misaka13514 commented 3 years ago

At some time all pixiv links will not work. (But it sometimes works fine.)

Now it seems to be working fine, if there are no problems in the next few days, I will close the issue.

*I tried almost the same operation before, and it worked normally after the deployment was completed, but it didn’t work after a while, so I came here.

Version info: Python

> cat runtime.txt 
python-3.9.5

requirements generated by pipreqs . --force

> cat requirements.txt 
PixivPy==3.5.11
pymongo==3.11.4
tenacity==5.1.5
PyDrive2==1.8.3
firebase_admin==5.0.0
aiohttp==3.7.4.post0
oauth2client==4.1.3
requests==2.25.1
Pybooru==4.2.2
aiogram==2.13
environs==9.3.2
motor==2.4.0
aiofiles==0.7.0
aiojobs==0.3.0
beautifulsoup4==4.9.3
dataclasses==0.8
tinydb==4.4.0

Remote reported an error, so I manually changed dataclasses==0.8 to dataclasses==0.6.

remote:        ERROR: Could not find a version that satisfies the requirement dataclasses==0.8 (from -r /tmp/build_23094bcc/requirements.txt (line 16)) (from versions: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6)

heroku stack: "heroku-20"

heroku remote detailed logs ```text remote: Building source: remote: remote: -----> Building on the Heroku-20 stack remote: -----> Using buildpack: heroku/python remote: -----> Python app detected remote: -----> Using Python version specified in runtime.txt remote: -----> Stack has changed from heroku-18 to heroku-20, clearing cache remote: -----> Requirements file has been changed, clearing cached dependencies remote: -----> Installing python-3.9.5 remote: -----> Installing pip 20.2.4, setuptools 47.1.1 and wheel 0.36.2 remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip remote: Collecting PixivPy==3.5.11 remote: Downloading PixivPy-3.5.11-py3-none-any.whl (17 kB) remote: Collecting pymongo==3.11.4 remote: Downloading pymongo-3.11.4-cp39-cp39-manylinux2014_x86_64.whl (518 kB) remote: Collecting tenacity==5.1.5 remote: Downloading tenacity-5.1.5-py2.py3-none-any.whl (34 kB) remote: Collecting PyDrive2==1.8.3 remote: Downloading PyDrive2-1.8.3-py2.py3-none-any.whl (31 kB) remote: Collecting firebase_admin==5.0.0 remote: Downloading firebase_admin-5.0.0-py3-none-any.whl (113 kB) remote: Collecting aiohttp==3.7.4.post0 remote: Downloading aiohttp-3.7.4.post0-cp39-cp39-manylinux2014_x86_64.whl (1.4 MB) remote: Collecting oauth2client==4.1.3 remote: Downloading oauth2client-4.1.3-py2.py3-none-any.whl (98 kB) remote: Collecting requests==2.25.1 remote: Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB) remote: Collecting Pybooru==4.2.2 remote: Downloading Pybooru-4.2.2-py2.py3-none-any.whl (22 kB) remote: Collecting aiogram==2.13 remote: Downloading aiogram-2.13-py3-none-any.whl (179 kB) remote: Collecting environs==9.3.2 remote: Downloading environs-9.3.2-py2.py3-none-any.whl (11 kB) remote: Collecting motor==2.4.0 remote: Downloading motor-2.4.0-py3-none-any.whl (54 kB) remote: Collecting aiofiles==0.7.0 remote: Downloading aiofiles-0.7.0-py3-none-any.whl (13 kB) remote: Collecting aiojobs==0.3.0 remote: Downloading aiojobs-0.3.0-py3-none-any.whl (10.0 kB) remote: Collecting beautifulsoup4==4.9.3 remote: Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB) remote: Collecting dataclasses==0.6 remote: Downloading dataclasses-0.6-py3-none-any.whl (14 kB) remote: Collecting tinydb==4.4.0 remote: Downloading tinydb-4.4.0-py3-none-any.whl (21 kB) remote: Collecting requests-toolbelt>=0.9.1 remote: Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB) remote: Collecting cloudscraper<1.2.50,>=1.0 remote: Downloading cloudscraper-1.2.48-py2.py3-none-any.whl (94 kB) remote: Collecting six>=1.9.0 remote: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) remote: Collecting PyYAML>=3.0 remote: Downloading PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl (630 kB) remote: Collecting pyOpenSSL>=19.1.0 remote: Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB) remote: Collecting google-api-python-client>=1.12.5 remote: Downloading google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB) remote: Collecting google-cloud-storage>=1.37.1 remote: Downloading google_cloud_storage-1.38.0-py2.py3-none-any.whl (103 kB) remote: Collecting cachecontrol>=0.12.6 remote: Downloading CacheControl-0.12.6-py2.py3-none-any.whl (19 kB) remote: Collecting google-cloud-firestore>=2.1.0; platform_python_implementation != "PyPy" remote: Downloading google_cloud_firestore-2.1.1-py2.py3-none-any.whl (209 kB) remote: Collecting google-api-core[grpc]<2.0.0dev,>=1.22.1; platform_python_implementation != "PyPy" remote: Downloading google_api_core-1.29.0-py2.py3-none-any.whl (93 kB) remote: Collecting multidict<7.0,>=4.5 remote: Downloading multidict-5.1.0-cp39-cp39-manylinux2014_x86_64.whl (151 kB) remote: Collecting attrs>=17.3.0 remote: Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB) remote: Collecting typing-extensions>=3.6.5 remote: Downloading typing_extensions-3.10.0.0-py3-none-any.whl (26 kB) remote: Collecting async-timeout<4.0,>=3.0 remote: Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB) remote: Collecting chardet<5.0,>=2.0 remote: Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB) remote: Collecting yarl<2.0,>=1.0 remote: Downloading yarl-1.6.3-cp39-cp39-manylinux2014_x86_64.whl (315 kB) remote: Collecting pyasn1-modules>=0.0.5 remote: Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) remote: Collecting rsa>=3.1.4 remote: Downloading rsa-4.7.2-py3-none-any.whl (34 kB) remote: Collecting pyasn1>=0.1.7 remote: Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) remote: Collecting httplib2>=0.9.1 remote: Downloading httplib2-0.19.1-py3-none-any.whl (95 kB) remote: Collecting certifi>=2017.4.17 remote: Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB) remote: Collecting urllib3<1.27,>=1.21.1 remote: Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB) remote: Collecting idna<3,>=2.5 remote: Downloading idna-2.10-py2.py3-none-any.whl (58 kB) remote: Collecting Babel>=2.8.0 remote: Downloading Babel-2.9.1-py2.py3-none-any.whl (8.8 MB) remote: Collecting python-dotenv remote: Downloading python_dotenv-0.17.1-py2.py3-none-any.whl (18 kB) remote: Collecting marshmallow>=2.7.0 remote: Downloading marshmallow-3.12.1-py2.py3-none-any.whl (47 kB) remote: Collecting soupsieve>1.2; python_version >= "3.0" remote: Downloading soupsieve-2.2.1-py3-none-any.whl (33 kB) remote: Collecting pyparsing>=2.4.7 remote: Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB) remote: Collecting cryptography>=3.2 remote: Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB) remote: Collecting google-auth-httplib2>=0.1.0 remote: Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB) remote: Collecting google-auth<2dev,>=1.16.0 remote: Downloading google_auth-1.30.1-py2.py3-none-any.whl (146 kB) remote: Collecting uritemplate<4dev,>=3.0.0 remote: Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB) remote: Collecting google-resumable-media<2.0dev,>=1.2.0 remote: Downloading google_resumable_media-1.3.0-py2.py3-none-any.whl (75 kB) remote: Collecting google-cloud-core<2.0dev,>=1.4.1 remote: Downloading google_cloud_core-1.6.0-py2.py3-none-any.whl (28 kB) remote: Collecting msgpack>=0.5.2 remote: Downloading msgpack-1.0.2-cp39-cp39-manylinux1_x86_64.whl (294 kB) remote: Collecting proto-plus>=1.10.0 remote: Downloading proto_plus-1.18.1-py3-none-any.whl (42 kB) remote: Collecting pytz remote: Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB) remote: Collecting googleapis-common-protos<2.0dev,>=1.6.0 remote: Downloading googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB) remote: Collecting protobuf>=3.12.0 remote: Downloading protobuf-3.17.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) remote: Collecting packaging>=14.3 remote: Downloading packaging-20.9-py2.py3-none-any.whl (40 kB) remote: Collecting grpcio<2.0dev,>=1.29.0; extra == "grpc" remote: Downloading grpcio-1.38.0-cp39-cp39-manylinux2014_x86_64.whl (4.3 MB) remote: Collecting cffi>=1.12 remote: Downloading cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl (406 kB) remote: Collecting cachetools<5.0,>=2.0.0 remote: Downloading cachetools-4.2.2-py3-none-any.whl (11 kB) remote: Collecting google-crc32c<2.0dev,>=1.0; python_version >= "3.5" remote: Downloading google_crc32c-1.1.2-cp39-cp39-manylinux2014_x86_64.whl (38 kB) remote: Collecting pycparser remote: Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB) remote: Installing collected packages: certifi, urllib3, chardet, idna, requests, requests-toolbelt, pyparsing, cloudscraper, PixivPy, pymongo, six, tenacity, pyasn1, pyasn1-modules, rsa, httplib2, oauth2client, PyYAML, pycparser, cffi, cryptography, pyOpenSSL, protobuf, googleapis-common-protos, pytz, packaging, cachetools, google-auth, grpcio, google-api-core, google-auth-httplib2, uritemplate, google-api-python-client, PyDrive2, google-crc32c, google-resumable-media, google-cloud-core, google-cloud-storage, msgpack, cachecontrol, proto-plus, google-cloud-firestore, firebase-admin, multidict, attrs, typing-extensions, async-timeout, yarl, aiohttp, Pybooru, Babel, aiogram, python-dotenv, marshmallow, environs, motor, aiofiles, aiojobs, soupsieve, beautifulsoup4, dataclasses, tinydb remote: Successfully installed Babel-2.9.1 PixivPy-3.5.11 PyDrive2-1.8.3 PyYAML-5.4.1 Pybooru-4.2.2 aiofiles-0.7.0 aiogram-2.13 aiohttp-3.7.4.post0 aiojobs-0.3.0 async-timeout-3.0.1 attrs-21.2.0 beautifulsoup4-4.9.3 cachecontrol-0.12.6 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 cloudscraper-1.2.48 cryptography-3.4.7 dataclasses-0.6 environs-9.3.2 firebase-admin-5.0.0 google-api-core-1.29.0 google-api-python-client-2.8.0 google-auth-1.30.1 google-auth-httplib2-0.1.0 google-cloud-core-1.6.0 google-cloud-firestore-2.1.1 google-cloud-storage-1.38.0 google-crc32c-1.1.2 google-resumable-media-1.3.0 googleapis-common-protos-1.53.0 grpcio-1.38.0 httplib2-0.19.1 idna-2.10 marshmallow-3.12.1 motor-2.4.0 msgpack-1.0.2 multidict-5.1.0 oauth2client-4.1.3 packaging-20.9 proto-plus-1.18.1 protobuf-3.17.2 pyOpenSSL-20.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pymongo-3.11.4 pyparsing-2.4.7 python-dotenv-0.17.1 pytz-2021.1 requests-2.25.1 requests-toolbelt-0.9.1 rsa-4.7.2 six-1.16.0 soupsieve-2.2.1 tenacity-5.1.5 tinydb-4.4.0 typing-extensions-3.10.0.0 uritemplate-3.0.1 urllib3-1.26.5 yarl-1.6.3 remote: -----> Discovering process types remote: Procfile declares types -> web remote: remote: -----> Compressing... remote: Done: 84.1M remote: -----> Launching... remote: Released v33 remote: https://misakapicbot.herokuapp.com/ deployed to Heroku remote: remote: Verifying deploy... done. ```
Misaka13514 commented 3 years ago

Everything seems fine, thank you @y-young !

KaidLi commented 3 years ago

@Misaka13514

Hello, I am experiencing the same problem as you. May I ask how you finally solved this problem?

2021-06-17 09:33:18,720 - bot - ERROR - Update caused error "requests POST https://oauth.secure.pixiv.net/auth/token error: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter."
Misaka13514 commented 3 years ago

In fact, I didn't solve it, but it solved itself: after a while, the error no longer appeared. I feel very strange about this. @KaidLi

Misaka13514 commented 3 years ago

My friend @Cn47mP also encountered this problem before. After she redeployed the robot , the problem seemed to be solved.

If the problem is still not solved after you try to upgrade the software and dependencies, provide your version information. I can reopen the issue for you, or maybe you can open a new one. @KaidLi

KaidLi commented 3 years ago

@Misaka13514 Thank you for your help! It worked fine after reinstalling.