Anorov / cloudflare-scrape

A Python module to bypass Cloudflare's anti-bot page.
MIT License
3.38k stars 459 forks source link

cfscrape hanging #198

Closed ohura88 closed 5 years ago

ohura88 commented 5 years ago

Trying to bypass Cloudflare on a site and the code just hangs. Looks to me like the request is timing out. Maybe something changed with the redirection?

Paste has the URL at the top and the html of the page, same NSFW link from last time: https://pastebin.com/Hp4tavEA

Here's my attempt at running the code

C:\Users\ohura>python Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

import cfscrape scraper = cfscrape.create_scraper() url = "http://www.javlibrary.com/en/?v=javliilj4y" contents = scraper.get(url).content.decode() Traceback (most recent call last): File "", line 1, in File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\sessions.py", line 546, in get return self.request('GET', url, kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init__.py", line 68, in request resp = self.solve_cf_challenge(resp, kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init.py", line 113, in solve_cf_challenge return self.request(method, redirect.headers["Location"], original_kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init__.py", line 68, in request resp = self.solve_cf_challenge(resp, kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init.py", line 113, in solve_cf_challenge return self.request(method, redirect.headers["Location"], original_kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init__.py", line 68, in request resp = self.solve_cf_challenge(resp, kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init__.py", line 113, in solve_cf_challenge return self.request(method, redirect.headers["Location"], original_kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init__.py", line 68, in request resp = self.solve_cf_challenge(resp, kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape__init__.py", line 113, in solve_cf_challenge return self.request(method, redirect.headers["Location"], **original_kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape\init.py", line 68, in request resp = self.solve_cf_challenge(resp, **kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape\init__.py", line 113, in solve_cf_challenge return self.request(method, redirect.headers["Location"], **original_kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape\init.py", line 68, in request resp = self.solve_cf_challenge(resp, **kwargs) File "C:\Users\ohura\AppData\Local\Programs\Python\Python36\lib\site-packages\cfscrape\init__.py", line 105, in solve_cf_challenge time.sleep(self.delay - (end_time - start_time)) # Cloudflare requires a delay before solving the challenge KeyboardInterrupt

armin884 commented 5 years ago

same error here....

gozames commented 5 years ago

same error here also

edimusxero commented 5 years ago

same issue, hangs forever until I manually kill it.

Andrflor commented 5 years ago

I'm guessing cloud-flare may have change the way they detect bots since i'm having the same issue and my code was just working fine yesterday..

EdmundMartin commented 5 years ago

Same issue here. I can see that the library is parsing a result from the challenge. It is also pulling out the redirect location after solving the challenge the initial time. It appears that there has been a change in the flow which has broken or that there a small variance in the expected challenge result.

An additional 's' value appears to be now being sent on resolving the challenge this is a hidden input on the form. Will make a pull request.

Anorov commented 5 years ago

Should be fixed in the latest version. Please run pip install -U cfscrape.