Always Captcha (with proxies)

[OSVFreeDom]

Please attempt to answer the following questions before submitting a new issue:

• What version of Cloudscraper are you using?
  • 4.1.2
• What version of Node.js are you using? (Please share the process information)
  • 10.16
• When did the problem start occurring? Today (2019/07/11)
• How often does the problem occur? All time
• What is the URL? Many URL including https://pro-src.com When I try to get an URL with proxy, I always get a captcha. I've try with Python cloudflare-scrape library and returning the body content without any problems. When I try with Python I use exactly the same proxy/User-Agent and same destination URL I've make 50 tries with each (node/Python) Python => 50 success Node => 0 success (50 captcha)

• [x] I have read the README. (Code examples)

Please share a minimal working code snippet that reproduces the problem.

Code snippet

```js var headers = { 'User-Agent':"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36", }; var cloudscraper = require('cloudscraper'); cloudscraper.defaults({proxy:"http://163.172.171.125:80", headers:headers}).get('https://pro-src.com').then(function(resp) { console.log(resp.body.toString('utf-8')); }).catch(function(err) { console.log(err); }); ``` ```python import cfscrape scraper = cfscrape.create_scraper() print (scraper.get("https://pro-src.com", proxies={'https': 'http://163.172.171.125:80'}, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36'}).content) ```

EDIT : I've made test with some differents proxy and User-Agent, each time I test with each script (node/Python)

Python return body all time and node never (allways captcha)

[ghost]

Hi @OSVFreeDom,

Thanks for taking the time to create a proper issue. It really helps. This seems to be the same problem as https://github.com/codemanki/cloudscraper/issues/229. Please read over that issue and let me know if I can be of more assistance.

As I mention in that issue, I'd really like to fix the problem but I don't currently have a concrete way of doing so that works for everybody.

Cheers

[OSVFreeDom]

Thanks for your answer. I've try some ciphers but no changes... Do you have a temporaly fix to force using good cipher that not trigger CAPTCHA ?

EDIT : The script run for more than 1 week with exactly the same version of Node/Cloudscraper and it's stop working today (no code change on the script/server)

[ghost]

If the problem was the cipher list, I could recommend something but that's not the exact problem. I've gone into more detail in the linked issue.

FWIW, I don't get a CAPTCHA when using https://nodejs.org/download/release/v10.16.0/node-v10.16.0-linux-x64.tar.gz

const cloudscraper = require('cloudscraper')
cloudscraper.get('https://pro-src.com').then(console.log)

OpenSSL may behave differently depending on the CPU. I've never had much problem with the TLS related Cloudflare updates, others have not fared as well. :(

Recompiling to expose the necessary OpenSSL functions would be one way of addressing this issue. I might get around to providing some instruction on that, no promises.

[OSVFreeDom]

I currently have this build of node and I get CAPTCHA... Thanks anyway

[ghost]

Yw and Sorry friend :(, please do check out the suggestions in that issue: https://github.com/codemanki/cloudscraper/issues/229#issuecomment-502481770

The OP did manage to get it working.

[OSVFreeDom]

I don't understand why Pyhton with same version of openssl don't have this issue

[ghost]

Depending on the OS, Python is more than likely using the system's version of OpenSSL. Node.js compiles with OpenSSL meaning it uses it's own version. So try a different build of Node.js.

[ghost]

Check node -p process.versions

[OSVFreeDom]

That's what I do, same bug with v12.6.0

[OSVFreeDom]

{
  node: '12.6.0',
  v8: '7.5.288.22-node.14',
  uv: '1.30.1',
  zlib: '1.2.11',
  brotli: '1.0.7',
  ares: '1.15.0',
  modules: '72',
  nghttp2: '1.38.0',
  napi: '4',
  llhttp: '1.1.4',
  http_parser: '2.8.0',
  openssl: '1.1.1c',
  cldr: '35.1',
  icu: '64.2',
  tz: '2019a',
  unicode: '12.1'
}

[OSVFreeDom]

It's only with using proxy, but same proxies on Python and Node but success rate is 100% for Pyhton and node 0%

[ghost]

Ah, so when not using a proxy everything works? If that's the case, this isn't likely to be a TLS problem. Well... Could you adjust the title? :smile:

Give this a go: https://github.com/codemanki/cloudscraper/issues/233#issuecomment-510342262

[OSVFreeDom]

I've read this issue and adjust the headers not working for me.

[ghost]

Ah, you're overriding the headers... duh

Here is your fixed snippet:

var cloudscraper = require('cloudscraper');
var headers = Object.assign(cloudscraper.defaultParams.headers, { /* modify UA here */ });

cloudscraper.defaults({proxy:"http://163.172.171.125:80", headers })
.get('https://pro-src.com').then(function(body) {
  console.log(body);
}).catch(function(err) {
  console.log(err);
});

[ghost]

I apologize for not running your snippet initially :/

[OSVFreeDom]

Same, trigger captcha (I try some proxy)... Python with same proxy not trigger captcha...

[ghost]

The reason you were experiencing a CAPTCHA was because of the way you were overriding the default headers that are needed to imitate the web browser. Please see the code snippet above.

[ghost]

Sorry for being short but this stuff is covered in the README and the examples. Good luck

[OSVFreeDom]

I think you don't understand... Your snippet with a lot of proxy not work (in Python yes) The big problem is : Yesterday => All working fine Today => nothing change (code/server etc..) and not working with any proxy (in Python it works too)

var target = "https://WEBSITE.COM"
var proxyURL = "http://IP:PORT"
var cloudscraper = require('cloudscraper').defaults({ resolveWithFullResponse: true, proxy: proxyURL });
    cloudscraper(target).then(function(resp) {
      //some code
    }).catch(function(err) {
      console.log(err.message) // always captcha
    })

Code tried with more than 10k proxies => 100% failed (5% dead proxy, 95% captcha) The same in Python => 93% success (5% dead proxy, 2% captcha)

PS : I don't override any headers (In the issue snippet it's for example, I should not have put this example sorry). My real code is this (see upper)

[ghost]

I can't reproduce the statistics you've given. I can't reproduce the problem with the proxy you shared. If you're not overriding the headers, I'm not quite sure what the problem could be but we're getting a lot reports about proxies not working so I don't think this is an isolated issue. Something has changed...

[OSVFreeDom]

Yes something has changed but nothing in my side on Cloudflare side perhaps. The script run a lot of time by day and without any reasons it's stop working (I don't have exact hour sorry)

[ghost]

@GoogleSites, @dpalade06, @izidan, @brunogaspar

Let's move all of the proxy related conversation here.

Does anybody else get the same results as @OSVFreeDom when running the script from https://github.com/codemanki/cloudscraper/issues/233#issuecomment-510342262

[OSVFreeDom]

In console.log you can see all request trigger captcha expect dead proxies

result.txt

```js fail: http://186.219.211.6:60926 fail: http://181.196.77.70:53281 fail: http://124.41.214.113:8080 fail: http://63.246.57.113:46312 fail: http://110.74.196.235:46140 fail: http://200.66.94.147:8080 fail: http://104.248.66.103:3128 fail: http://142.93.240.41:3128 fail: http://138.197.216.112:8118 fail: http://91.217.63.121:40079 fail: http://66.116.87.134:8080 fail: http://187.189.73.66:999 fail: http://169.255.5.174:8080 fail: http://171.100.86.113:8080 fail: http://92.115.134.57:8080 fail: http://86.123.166.109:8080 fail: http://95.31.130.96:37138 fail: http://104.152.45.45:80 fail: http://139.5.73.11:8080 fail: http://117.54.250.2:60791 fail: http://151.106.8.232:8080 fail: http://70.82.156.40:8888 fail: http://114.110.22.90:43221 fail: http://97.72.106.202:87 fail: http://89.28.53.42:8080 fail: http://176.115.197.118:8080 fail: http://12.189.125.134:8080 fail: http://13.58.37.5:3128 fail: http://118.175.244.91:30538 fail: http://212.237.30.203:8888 fail: http://78.187.100.171:8888 ```

console.log

``` http://142.93.240.41:3128 x RequestError: Error: tunneling socket could not be established, statusCode=403 http://104.152.45.45:80 x RequestError: Error: tunneling socket could not be established, statusCode=400 http://104.248.66.103:3128 x RequestError: Error: tunneling socket could not be established, statusCode=403 http://118.175.244.91:30538 x CaptchaError: captcha Result: http://91.217.63.121:40079 x CaptchaError: captcha Result: http://200.66.94.147:8080 x CaptchaError: captcha Result: http://63.246.57.113:46312 x CaptchaError: captcha Result: http://138.197.216.112:8118 x RequestError: Error: tunneling socket could not be established, cause=socket hang up http://151.106.8.232:8080 x CaptchaError: captcha Result: http://86.123.166.109:8080 x CaptchaError: captcha Result: http://171.100.86.113:8080 x CaptchaError: captcha Result: http://186.219.211.6:60926 x CaptchaError: captcha Result: http://78.187.100.171:8888 x CaptchaError: captcha Result: http://139.5.73.11:8080 x CaptchaError: captcha Result: http://176.115.197.118:8080 x CaptchaError: captcha Result: http://187.189.73.66:999 x CaptchaError: captcha Result: http://97.72.106.202:87 x RequestError: Error: tunneling socket could not be established, statusCode=514 http://92.115.134.57:8080 x RequestError: Error: tunneling socket could not be established, cause=read ECONNRESET http://66.116.87.134:8080 x CaptchaError: captcha Result: http://89.28.53.42:8080 x RequestError: Error: tunneling socket could not be established, cause=read ECONNRESET http://169.255.5.174:8080 x CaptchaError: captcha Result: http://181.196.77.70:53281 x CaptchaError: captcha Result: http://212.237.30.203:8888 x RequestError: Error: tunneling socket could not be established, statusCode=503 http://110.74.196.235:46140 x RequestError: Error: tunneling socket could not be established, cause=read ECONNRESET ```

[brunogaspar]

Node version: v11.12.0

result.txt

``` fail: http://144.217.49.233:3128 fail: http://124.133.230.254:80 fail: http://190.60.69.42:8080 fail: http://114.252.203.62:8060 fail: http://200.111.182.6:443 fail: http://103.196.233.199:8080 fail: http://178.92.9.210:3128 fail: http://41.223.66.73:8080 fail: http://82.117.252.123:8080 fail: http://169.255.126.211:8080 fail: http://190.60.69.114:8080 fail: http://190.60.69.162:8080 fail: http://200.123.2.171:3128 fail: http://41.216.163.230:8080 fail: http://46.63.44.170:8080 fail: http://179.57.30.10:8080 fail: http://109.86.190.193:8080 fail: http://190.60.69.194:8080 fail: http://195.80.140.212:8081 fail: http://119.110.204.62:80 fail: http://27.147.176.110:8080 fail: http://85.159.2.171:8080 fail: http://105.235.197.22:8080 fail: http://114.199.110.58:47428 fail: http://41.204.232.218:8080 fail: http://175.139.218.9:8080 fail: http://106.2.238.2:3128 fail: http://60.205.188.24:3128 fail: http://85.198.142.186:8081 fail: http://91.224.156.197:8080 fail: http://105.235.197.190:8080 ```

console.log

``` http://144.217.49.233:3128 x RequestError: Error: tunneling socket could not be established, cause=connect ECONNREFUSED 144.217.49.233:3128 http://124.133.230.254:80 x RequestError: Error: tunneling socket could not be established, statusCode=403 http://195.80.140.212:8081 x RequestError: Error: tunneling socket could not be established, statusCode=400 http://106.2.238.2:3128 x RequestError: Error: tunneling socket could not be established, statusCode=400 http://60.205.188.24:3128 x RequestError: Error: tunneling socket could not be established, statusCode=400 http://46.63.44.170:8080 x CaptchaError: captcha Result: http://114.252.203.62:8060 x RequestError: Error: tunneling socket could not be established, cause=Parse Error http://27.147.176.110:8080 x CaptchaError: captcha Result: http://114.199.110.58:47428 x CaptchaError: captcha Result: http://200.111.182.6:443 x CaptchaError: captcha Result: http://190.60.69.42:8080 x CaptchaError: captcha Result: http://179.57.30.10:8080 x CaptchaError: captcha Result: http://190.60.69.162:8080 x CaptchaError: captcha Result: http://169.255.126.211:8080 x CaptchaError: captcha Result: http://85.198.142.186:8081 x CaptchaError: captcha Result: http://175.139.218.9:8080 x CaptchaError: captcha Result: http://190.60.69.194:8080 x RequestError: Error: tunneling socket could not be established, cause=read ECONNRESET http://91.224.156.197:8080 x CaptchaError: captcha Result: [freeproxylists] Navigation Timeout Exceeded: 20000ms exceeded [xroxy] Navigation Timeout Exceeded: 20000ms exceeded http://41.216.163.230:8080 x CaptchaError: captcha Result: [hidemyname] waiting for selector ".proxy__t tbody tr" failed: timeout 20000ms exceeded http://119.110.204.62:80 x RequestError: Error: tunneling socket could not be established, cause=socket hang up http://105.235.197.190:8080 x CaptchaError: captcha Result: http://105.235.197.22:8080 x CaptchaError: captcha Result: http://190.60.69.114:8080 x CaptchaError: captcha Result: http://82.117.252.123:8080 x CaptchaError: captcha Result: http://178.92.9.210:3128 x CaptchaError: captcha Result: http://200.123.2.171:3128 x CaptchaError: captcha Result: [sockslist] waiting for selector ".proxytbl tbody tr:not(:first-child)" failed: timeout 20000ms exceeded http://103.196.233.199:8080 x CaptchaError: captcha Result: ```

I guess i have the exact same behaviour.

Haven't spent much time figuring things out yet, had other stuff to work on.

[ghost]

wget https://nodejs.org/download/release/v11.12.0/node-v11.12.0-linux-x64.tar.gz
tar xzvf node-v11.12.0-linux-x64.tar.gz
./node-v11.12.0-linux-x64/bin/node issue-233.js

results.txt

``` fail: http://178.63.234.254:8080 pass: http://154.73.141.109:53972 pass: http://188.136.222.1:9090 fail: http://46.201.219.126:8080 pass: http://82.99.204.30:8080 fail: http://113.53.225.66:8080 fail: http://195.78.93.28:3128 fail: http://41.78.243.250:53281 fail: http://116.0.54.30:8080 fail: http://46.151.83.231:443 fail: http://94.153.223.210:8888 fail: http://212.25.36.55:8085 fail: http://42.201.161.126:8080 fail: http://113.161.68.146:8080 fail: http://41.78.243.242:53281 pass: http://125.209.78.118:8080 fail: http://37.187.4.81:8118 pass: http://154.72.76.246:8080 fail: http://94.240.221.20:8080 pass: http://195.9.162.186:8080 fail: http://202.142.158.114:8080 fail: http://195.34.241.85:8080 pass: http://91.137.140.89:8082 fail: http://91.210.136.202:8080 fail: http://202.182.57.10:8080 fail: http://195.242.179.23:8080 fail: http://212.237.30.203:3128 fail: http://110.49.69.66:8080 fail: http://5.79.113.168:3128 pass: http://183.88.212.184:8080 fail: http://195.24.53.194:80 ```

The only thing that I imagine to be different between our systems is the TLS because OpenSSL defaults might vary depending on CPU. It's the best explanation that I have at the moment given everything that CF has been doing with TLS lately. It only really makes sense if CF is performing additional checks when they encounter a known proxy. Because you guys don't get CAPTCHA when not using proxies, right???

Interesting though, I do seem to get more CAPTCHA responses on this version of node. Shooting in the dark, would you guys try a couple of different ciphers as mentioned here: https://github.com/codemanki/cloudscraper/issues/229#issuecomment-502483820

[brunogaspar]

Because you guys don't get CAPTCHA when not using proxies, right???

Yup, i rarely get captcha when i don't use proxies.

Interesting though, I do seem to get more CAPTCHA responses on this version of node.

I can try different versions of Node later today and see where it lands me.

Shooting in the dark, would you guys try a couple of different ciphers as mentioned here: #229 (comment)

I tried that, kinda the same result unfortunately.

[ghost]

I'm not having much luck on my end... Reading https://tools.ietf.org/html/rfc8446#section-8.3 through https://tools.ietf.org/html/rfc8446#section-9.3 and wondering if CF is timing the TLS negotiation as a way to spot proxies.

Each of the following does yield passing results:

~/Downloads/node-v11.12.0-linux-x64/bin/node --tls-cipher-list='ECDHE-ECDSA-AES128-GCM-SHA256' issue-233.js
~/Downloads/node-v11.12.0-linux-x64/bin/node --tls-cipher-list='ECDHE-ECDSA-AES256-GCM-SHA384' issue-233.js
~/Downloads/node-v11.12.0-linux-x64/bin/node --tls-cipher-list='ECDHE-ECDSA-CHACHA20-POLY1305' issue-233.js

[OSVFreeDom]

It's work for me ! Thanks 👍

[brunogaspar]

These are my results

node --tls-cipher-list='ECDHE-ECDSA-AES128-GCM-SHA256' issue-233.js > results.txt

result.txt

```sh pass: http://95.175.14.25:8080 fail: http://91.205.218.32:80 fail: http://159.138.22.112:80 pass: http://202.91.40.26:53281 pass: http://91.239.180.202:8080 fail: http://31.193.90.243:8080 pass: http://176.103.45.24:8080 pass: http://91.224.156.197:8080 fail: http://117.191.11.108:80 fail: http://37.247.209.179:8080 fail: http://58.162.210.160:53281 pass: http://177.74.112.162:8080 fail: http://91.233.158.143:8080 fail: http://162.211.126.220:443 fail: http://170.81.247.47:3128 pass: http://130.117.173.182:47278 pass: http://203.246.112.133:3128 fail: http://36.71.150.113:80 fail: http://202.182.160.50:3128 pass: http://81.163.57.121:41258 fail: http://195.80.140.212:8081 pass: http://81.17.131.59:8080 fail: http://60.205.188.24:3128 fail: http://95.47.116.73:80 fail: http://213.222.34.200:53281 fail: http://202.51.98.131:80 pass: http://94.75.125.119:8080 fail: http://1.20.100.227:57396 fail: http://96.30.53.124:3128 fail: http://104.131.214.218:80 pass: http://193.110.115.220:3128 ```

---

node --tls-cipher-list='ECDHE-ECDSA-AES256-GCM-SHA384' issue-233.js > results.txt

result.txt

```sh fail: http://95.175.14.25:8080 fail: http://91.205.218.32:80 fail: http://159.138.22.112:80 pass: http://202.91.40.26:53281 pass: http://91.239.180.202:8080 fail: http://31.193.90.243:8080 pass: http://176.103.45.24:8080 pass: http://91.224.156.197:8080 fail: http://117.191.11.108:80 fail: http://37.247.209.179:8080 fail: http://58.162.210.160:53281 pass: http://177.74.112.162:8080 fail: http://91.233.158.143:8080 fail: http://162.211.126.220:443 pass: http://170.81.247.47:3128 pass: http://130.117.173.182:47278 pass: http://203.246.112.133:3128 fail: http://36.71.150.113:80 fail: http://202.182.160.50:3128 fail: http://81.163.57.121:41258 fail: http://195.80.140.212:8081 pass: http://81.17.131.59:8080 fail: http://60.205.188.24:3128 fail: http://95.47.116.73:80 fail: http://213.222.34.200:53281 fail: http://202.51.98.131:80 fail: http://94.75.125.119:8080 fail: http://1.20.100.227:57396 pass: http://96.30.53.124:3128 fail: http://104.131.214.218:80 pass: http://193.110.115.220:3128 ```

---

node --tls-cipher-list='ECDHE-ECDSA-CHACHA20-POLY1305' issue-233.js > results.txt

result.txt

```sh pass: http://1.20.99.122:8080 fail: http://125.26.7.83:8080 fail: http://58.96.153.149:8080 fail: http://185.14.6.134:8080 pass: http://182.93.75.105:8080 fail: http://61.19.151.170:8080 fail: http://195.138.92.152:35245 pass: http://114.7.16.242:8080 pass: http://119.46.146.58:8080 fail: http://164.115.22.45:8080 pass: http://181.198.242.220:8080 fail: http://186.4.137.105:8080 fail: http://181.211.38.138:8080 pass: http://45.248.138.150:8080 pass: http://124.41.214.113:8080 fail: http://36.92.22.70:8080 pass: http://159.192.97.49:8080 fail: http://202.70.84.1:8080 fail: http://116.212.129.134:8080 fail: http://54.39.139.2:80 pass: http://45.120.119.55:59565 fail: http://182.93.95.146:8080 fail: http://103.75.148.102:8080 pass: http://93.127.114.60:3129 fail: http://85.93.177.178:80 fail: http://52.124.6.146:40834 fail: http://83.211.42.202:80 pass: http://96.9.66.112:8080 pass: http://96.9.67.84:8080 pass: http://193.110.115.220:3128 fail: http://123.1.170.138:3128 ```

---

Without any proxy (just removed the proxy from being passed on the cloudscraper)

result.txt

```sh pass: http://144.217.49.233:3128 pass: http://124.133.230.254:80 pass: http://1.20.99.122:8080 pass: http://125.26.7.83:8080 pass: http://58.96.153.149:8080 pass: http://185.14.6.134:8080 pass: http://182.93.75.105:8080 pass: http://61.19.151.170:8080 pass: http://195.138.92.152:35245 pass: http://114.7.16.242:8080 pass: http://119.46.146.58:8080 pass: http://164.115.22.45:8080 pass: http://181.198.242.220:8080 pass: http://186.4.137.105:8080 pass: http://181.211.38.138:8080 pass: http://45.248.138.150:8080 pass: http://124.41.214.113:8080 pass: http://36.92.22.70:8080 pass: http://159.192.97.49:8080 pass: http://202.70.84.1:8080 pass: http://116.212.129.134:8080 pass: http://54.39.139.2:80 pass: http://45.120.119.55:59565 pass: http://182.93.95.146:8080 pass: http://103.75.148.102:8080 pass: http://93.127.114.60:3129 pass: http://85.93.177.178:80 pass: http://52.124.6.146:40834 pass: http://83.211.42.202:80 pass: http://96.9.66.112:8080 pass: http://96.9.67.84:8080 ```

---

So i guess this is more tricky than it looks heh

[ghost]

Those are the results that I'd expect to see from random public proxies if everything is working properly. What was confusing about this is that I jumped to the conclusion of it being TLS related but then dismissed it because normally, you'd always be getting a CAPTCHA regardless of whether or not you're using a proxy. The only reason I revisited it was because nothing else made sense.

What changed? It seems to be the case that Cloudflare is only performing some TLS based checks when they detect a proxy otherwise you'd get a CAPTCHA without the proxy. The reason I've came to that conclusion is because the TLS should be the same even when using a proxy.

So the problem is the cipher list? No, I don't think so. I'm using the same cipher list on the same version of node, same OpenSSL version, etc. I don't get the CAPTCHA when using a proxy or otherwise.

What is the problem? The signature algorithms extension is different on your systems. Changing the cipher list has the side effect of modifying the signature algorithms. It's a side effect. We don't have a way to directly tell OpenSSL which signature algorithms to use because that function isn't exposed.

Solution? Find which ciphers we need to omit from the cipher list and/or only specify a single cipher to rule everything else out.

Ahem, update the library? We're already filtering some ciphers out of the list. Removing good ciphers isn't a good idea since this problem doesn't affect everybody. Removing them would affect compatibility with at least non-Cloudflare sites and is very hacky since the cipher list isn't directly responsible.

Who is affected? Not everybody. I'm not really sure what conditions are exactly responsible for causing OpenSSL to behave differently. It's probably CPU features. For those who don't have this problem with cfscrape(python), you might try to find a version of node that was compiled with the same version of OpenSSL as python is using.

If the suggestions above don't work for any proxies, lmk in a new issue.

[ghost]

This is the relevant Node.js issue: https://github.com/nodejs/node/issues/24818