search

Anorov / cloudflare-scrape

A Python module to bypass Cloudflare's anti-bot page.
MIT License
3.4k stars 462 forks source link

Fixed cf new challenge solution #373

Open Sraq-Zit opened 4 years ago

Sraq-Zit commented 4 years ago

There was mainly 3 problems:

  1. Value of jschl_vc token was taken from commented input element instead of the actual one
  2. The value of the div element with its ID as the variable k is split into multiple divs and numbered at the end of the ID for each div
  3. node: setInterval not defined. I removed that part from js code

Tested on:

EDIT As of yesterday the tested websites are no more working, due to the new cf challenge. The challenge lies within the script at the path that goes like example.com/cdn-cgi/challenge-platform/orchestrate/jsch/v1 Once the script is loaded and executed, the function _cf_chl_enter() is called.

I will try and see if I can automate the same process in python

mcronce commented 4 years ago

I'm getting a failure when I install from your fork:

ERROR:root:Error executing Cloudflare IUAM Javascript. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
ERROR:root:'https://www.iptorrents.com/login.php' returned an error. Could not collect tokens.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 386, in get_tokens
    resp = scraper.get(url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 204, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 331, in solve_challenge
    raise subprocess.CalledProcessError(node.returncode, "node -e ...", stderr)
subprocess.CalledProcessError: Command 'node -e ...' returned non-zero exit status 1

I'm going to try to manually run node and see if its output provides any clue as to why it's bombing out, will report back with results

Sraq-Zit commented 4 years ago

Alright !

By the way, I tried it and it worked fine with me

image

pilardo commented 4 years ago

@Sraq-Zit i'm newbie. how can i update my local file?

Sraq-Zit commented 4 years ago

@Sraq-Zit i'm newbie. how can i update my local file?

Reinstall it through this fork

pip install https://github.com/Sraq-Zit/cloudflare-scrape/archive/master.zip

andress134 commented 4 years ago

I'm getting a failure when I install from your fork:

ERROR:root:Error executing Cloudflare IUAM Javascript. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
ERROR:root:'https://www.iptorrents.com/login.php' returned an error. Could not collect tokens.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 386, in get_tokens
    resp = scraper.get(url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 129, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 204, in solve_cf_challenge
    answer, delay = self.solve_challenge(body, domain)
  File "/usr/lib/python2.7/site-packages/cfscrape/__init__.py", line 331, in solve_challenge
    raise subprocess.CalledProcessError(node.returncode, "node -e ...", stderr)
subprocess.CalledProcessError: Command 'node -e ...' returned non-zero exit status 1

I'm going to try to manually run node and see if its output provides any clue as to why it's bombing out, will report back with results

I see cloudflare has updated again the default challenge for uam

F170x commented 4 years ago

Doesn't work with: https://www.pexels.com/search/man/?format=js&seed=&page=2&type= I don't know how to cloudflare detect between browser incognit mode and a python script, maybe with a difference in the headers or in the ssl protocol version?

jaeakle commented 4 years ago

Did Cloudflare change again? Im getting the error with this line as usual: r"s,t,o,p, b,r,e,a,k,i,n,g,cpo,f.+?\r?\n[\s\S]+?a\.value\s*=.+?)\r?\n"

rnyPlanet commented 4 years ago

works today?

andress134 commented 4 years ago

works today?

no

rnyPlanet commented 4 years ago

@chanind @Sraq-Zit can you help update?

hellcoder420 commented 3 years ago

can someone please continue the work of this pr

ngooty commented 3 years ago

Is there a solution for this problem. am stuck since a month to resolve this.

rnyPlanet commented 3 years ago

Is there a solution for this problem. am stuck since a month to resolve this.

Wow. and how are you getting on with the solution? I also tried but I have little knowledge ..

ngooty commented 3 years ago

I have not got any solution or workaround. still stuck with the problem

thanks, Nag

On Mon, Dec 28, 2020 at 2:15 PM Stanislav notifications@github.com wrote:

Is there a solution for this problem. am stuck since a month to resolve this.

Wow. and how are you getting on with the solution? I also tried but I have little knowledge ..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Anorov/cloudflare-scrape/pull/373#issuecomment-751635093, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANN2TRNDXX22OJGVOIWBMN3SXBASRANCNFSM4ODFEWRA .

ngooty commented 3 years ago

Hi,

I got a workaround for the problem.

get signed up to "scrapingant.com" and use its API (it will produce unique key) to scape the websites behind cloud flare.

Thanks, Nag

On Mon, Dec 28, 2020 at 2:51 PM Nagaraju Gooty ngooty@gmail.com wrote:

I have not got any solution or workaround. still stuck with the problem

thanks, Nag

On Mon, Dec 28, 2020 at 2:15 PM Stanislav notifications@github.com wrote:

Is there a solution for this problem. am stuck since a month to resolve this.

Wow. and how are you getting on with the solution? I also tried but I have little knowledge ..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Anorov/cloudflare-scrape/pull/373#issuecomment-751635093, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANN2TRNDXX22OJGVOIWBMN3SXBASRANCNFSM4ODFEWRA .

andress134 commented 3 years ago

@Sraq-Zit is posible to fix v1 challenge again? thanks

hirios commented 2 years ago

+ 1