Recaptcha page always being sent.

krusynth commented 4 years ago

Please attempt to answer the following questions before submitting a new issue:

What version of Cloudscraper are you using? 4.5.0
What version of Node.js are you using? 12.12.0
When did the problem start occurring? About two weeks ago
How often does the problem occur? Every time
What is the URL? Any page on CurseForge, e.g. https://www.curseforge.com/minecraft/mc-mods/minecolonies/files/all?sort=-game-version&page=1
Are there any similar issues? (Please share the links) #307 #301

[X] I have read the README. (Code examples)

Please share a minimal working code snippet that reproduces the problem.

Code snippet

```js cloudscraper.get('https://www.curseforge.com/minecraft/mc-mods/minecolonies/files/all?sort=-game-version&page=1'); ```

Error Dump

```js CaptchaError: captcha at validateResponse (/Users/krues8dr/Projects/Minecraft/lazuli/node_modules/cloudscraper/index.js:273:11) at onCloudflareResponse (/Users/krues8dr/Projects/Minecraft/lazuli/node_modules/cloudscraper/index.js:222:5) at onRequestResponse (/Users/krues8dr/Projects/Minecraft/lazuli/node_modules/cloudscraper/index.js:205:5) at Request. (/Users/krues8dr/Projects/Minecraft/lazuli/node_modules/cloudscraper/index.js:149:7) at Object.onceWrapper (events.js:300:26) at Request.emit (events.js:210:5) at Request. (/Users/krues8dr/Projects/Minecraft/lazuli/node_modules/request/request.js:1161:10) at Request.emit (events.js:210:5) at Gunzip. (/Users/krues8dr/Projects/Minecraft/lazuli/node_modules/request/request.js:1083:12) at Object.onceWrapper (events.js:299:28) at Gunzip.emit (events.js:215:7) at endReadableNT (_stream_readable.js:1183:12) at processTicksAndRejections (internal/process/task_queues.js:80:21) { name: 'CaptchaError', message: 'captcha' } ```

I have started consistently receiving the Recaptcha page on every request via cloudscraper, even when specifying the TLSv1.2 cipher. This started occurring at about the same time as the other Recaptcha issues ( #307 #301 ) were reported so this leads me to suspect something may have changed with Cloudflare's detection algorithm?

codemanki commented 4 years ago

@krues8dr thanks for reporting this. For some reason I can't replicate this behaviour neither from my machine or multiple VPS in other regions. Did you notice any special behaviour or scenarios where recaptcha persists? Any findings would be helpful

codemanki commented 4 years ago

Should be fixed in 4.6.0. Please update and try again

krusynth commented 4 years ago

I updated but I'm still receiving the same error. I'll try to stand up a new server this weekend and retry in case this is an IP blocking issue or something.

codemanki commented 4 years ago

@krusynth it is really hard for me to repro these kind of issues, as from my IP your url works fine :( So it would be a good idea if you could try from a different IP address first

krusynth commented 4 years ago

Ok, so I tried this on a new server and it's working there. Though I'm still not sure why it won't work on my personal laptop, I'll have to create a proxy server I guess. Anyway, thank you for the great tool and all the help!

sinisterdev commented 4 years ago

Got the same issue right now, visiting the page on my browser gives me no issues but when I try with cloud scraper I get a captcha page

krusynth commented 4 years ago

I'm now getting this error on any new server on DigitalOcean I stand up as well. Something squirrelly has changed with their algorithm.

codemanki / cloudscraper

Recaptcha page always being sent. #314