cfscrape.get(url).content returns "Attention Required!

JafarAbbas33 commented 4 years ago

Please confirm the following statements and check the boxes before creating an issue:

[X ] I've upgraded cfscrape with pip install -U cfscrape
[ ] I'm using Node version 10 or higher
[X ] The site protection I'm having issues with is from Cloudflare
[X ] I'm not using Tor, a VPN, or an anonymizing proxy

Python version number

Run python --version and paste the output below: -> Python 3.8.2

cfscrape version number

Run pip show cfscrape and paste the output below: -> Name: cfscrape Version: 2.1.1 Summary: A simple Python module to bypass Cloudflare's anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information. Home-page: https://github.com/Anorov/cloudflare-scrape Author: Anorov Author-email: anorov.vorona@gmail.com License: UNKNOWN Location: c:\users\jafar33\appdata\local\programs\python\python38-32\lib\site-packages Requires: requests Required-by:

Code snippet involved with the issue

scraper = cfscrape.create_scraper()
url = "https://www.pexels.com/search/man/?format=js&seed=&page=2&type="
htmltext = scraper.get(url).content

URL of the Cloudflare-protected page

https://www.pexels.com/search/man/?format=js&seed=&page=2&type=

caipenpen commented 4 years ago

Note: This only works when regular Cloudflare anti-bots is enabled (the "Checking your browser before accessing..." loading page). If there is a reCAPTCHA challenge, you're out of luck. Thankfully, the JavaScript check page is much more common.

JafarAbbas33 commented 4 years ago

But if you go to Pexels then you can see there is no captcha required. Then why is that happening? (Note: I have previously used cloudscraper which used to work but stopped working recently so I thought maybe i should move on to another library.)

andress134 commented 4 years ago

something know any lib that can bypass new captcha challenge?

JafarAbbas33 commented 4 years ago

@andress134 You having the same problem?

andress134 commented 4 years ago

@andress134 You having the same problem?

Yes, cfscrape can't bypass captcha, just iuam v1 challenge. On this moment i think nobody (public lib) like cloudscraper, no one can't bypass new challenge

caipenpen commented 4 years ago

But if you go to Pexels then you can see there is no captcha required. Then why is that happening? (Note: I have previously used cloudscraper which used to work but stopped working recently so I thought maybe i should move on to another library.)

No , it have . Yesterday , I check and it need me resolve captcha ......... Please try again today , I see it already close the captcha

JafarAbbas33 commented 4 years ago

I don't have to solve a captcha. I think it is because of some cookie. I took a snapshot of cookies used by the session but none of them made any sense.

caipenpen commented 4 years ago

I don't have to solve a captcha. I think it is because of some cookie. I took a snapshot of cookies used by the session but none of them made any sense.

What do you need to do this website ? Ddos it ???

JafarAbbas33 commented 4 years ago

@caipenpen Cmon, no man XD. I want to automate getting some photos from a search result.

andress134 commented 4 years ago

Looking for some user who can make a captcha bypass using public lib like 2captcha, anticaptcha or who can fix cloudscraper(nodejs) captcha function I can pay 200$

caipenpen commented 4 years ago

@caipenpen Cmon, no man XD. I want to automate getting some photos from a search result.

Why don't you use API ?

JafarAbbas33 commented 4 years ago

@caipenpen To be truthful, I have no idea why that didn't occur to me. Thanks a lot! But still this library needs to be fixed. Right?

caipenpen commented 4 years ago

@caipenpen To be truthful, I have no idea why that didn't occur to me. Thanks a lot! But still this library needs to be fixed. Right?

Yes , but we use it free and the cloudflare is update day by day , we can't rush the developer . I think you should try use api of pexels , I see it have search option

JafarAbbas33 commented 4 years ago

Yes you are right. And thanks to you, I can again fetch photos from Pexels (ofcourse I tried and it worked like a charm). So I will leave this issue opened as it has to be solved anyway. Thanks everyone.

F170x commented 3 years ago

I have the same issue with another site, when you open the url from your browser (chrome, firefox, safari, etc) works perfect without captcha required. Also in incognit mode. But when you try to make a request with this library I get the captcha error.

I think that cloudflare maybe detect anything in the headers (sort of headers, upper or lower cases, etc) or in the ssl protocol (TLS version, cypher, etc)

Anyone have updates about this?

Anorov / cloudflare-scrape