search

Anorov / cloudflare-scrape

A Python module to bypass Cloudflare's anti-bot page.
MIT License
3.35k stars 458 forks source link

ValueError: Unable to identify Cloudflare IUAM Javascript on website. #386

Open spfeifer222 opened 4 years ago

spfeifer222 commented 4 years ago

Version:

In [11]: !pip install -U cfscrape                                                                                                                                                                                  
Requirement already up-to-date: cfscrape in ./bin/miniconda/envs/SCRAP/lib/python3.7/site-packages (2.1.1)
Requirement already satisfied, skipping upgrade: requests>=2.23.0 in ./bin/miniconda/envs/SCRAP/lib/python3.7/site-packages (from cfscrape) (2.24.0)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./bin/miniconda/envs/SCRAP/lib/python3.7/site-packages (from requests>=2.23.0->cfscrape) (1.25.8)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in ./bin/miniconda/envs/SCRAP/lib/python3.7/site-packages (from requests>=2.23.0->cfscrape) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in ./bin/miniconda/envs/SCRAP/lib/python3.7/site-packages (from requests>=2.23.0->cfscrape) (2.9)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./bin/miniconda/envs/SCRAP/lib/python3.7/site-packages (from requests>=2.23.0->cfscrape) (2020.6.20)

Code:

from cfscrape import create_scraper

url = 'https://soap2day.to'   # also tried: 'https://www.soap2day.to'

sc = create_scraper(url).content

Error:


ValueError: Unable to identify Cloudflare IUAM Javascript on website. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
shawnngtq commented 4 years ago

@spfeifer222, did you find any solution? Or new package / library that works?

janwendt commented 3 years ago

@spfeifer222, did you find any solution? Or new package / library that works?

I went for scrapy-splash and use it that way: https://github.com/scrapy-plugins/scrapy-splash/issues/92#issuecomment-680050082