codemanki / cloudscraper

--DEPRECATED -- 🛑 🛑 Node.js library to bypass cloudflare's anti-ddos page
MIT License
602 stars 140 forks source link

1020, Access Denied (Custom Firewall Rules) #343

Open ChesterDenn opened 4 years ago

ChesterDenn commented 4 years ago

I cannot access https://www.mabbank.com/ today. my server is located in Singapore. until today, it's working well.

REQUEST { requester:
   { [Function: request]
     get: [Function],
     head: [Function],
     options: [Function],
     post: [Function],
     put: [Function],
     patch: [Function],
     del: [Function],
     delete: [Function],
     jar: [Function],
     cookie: [Function],
     defaults: [Function],
     forever: [Function],
     Request:
      { [Function: Request]
        super_: [Function],
        debug: true,
        defaultProxyHeaderWhiteList: [Array],
        defaultProxyHeaderExclusiveList: [Array] },
     initParams: [Function: initParams],
     debug: [Getter/Setter],
     bindCLS: [Function: RP$bindCLS] },
  jar:
   RequestJar {
     _jar: CookieJar { enableLooseMode: true, store: { idx: {} } } },
  headers:
   { Host: Symbol(host),
     Connection: 'keep-alive',
     'Upgrade-Insecure-Requests': '1',
     'User-Agent':
      'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36',
     Accept:
      'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
     'Accept-Language': 'en-US,en;q=0.8',
     'Accept-Encoding': 'gzip, deflate, sdch, br' },
  cloudflareMaxTimeout: 30000,
  followAllRedirects: true,
  challengesToSolve: 3,
  decodeEmails: false,
  gzip: true,
  agentOptions:
   { ciphers:
      'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA:!ECDHE+SHA:!AES128-SHA' },
  uri: 'https://www.mabbank.com/',
  method: 'GET',
  realEncoding: 'utf8',
  encoding: null,
  callback: [Function: RP$callback],
  transform: undefined,
  simple: true,
  resolveWithFullResponse: false,
  transform2xxOnly: false }
REQUEST make request https://www.mabbank.com/
REQUEST onRequestResponse https://www.mabbank.com/ 503 { date: 'Wed, 22 Apr 2020 05:46:48 GMT',
  'content-type': 'text/html; charset=UTF-8',
  'transfer-encoding': 'chunked',
  connection: 'close',
  'x-frame-options': 'SAMEORIGIN',
  'set-cookie':
   [ '__cfduid=dcb46a322884336e07ec1218bb44d86821587534408; expires=Fri, 22-May-20 05:46:48 GMT; path=/; domain=.mabbank.com; HttpOnly; SameSite=Lax' ],
  'cache-control': 'no-cache',
  'expect-ct':
   'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  vary: 'Accept-Encoding',
  'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
  'x-content-type-options': 'nosniff',
  server: 'cloudflare',
  'cf-ray': '587d0be68ee7e233-SIN',
  'cf-request-id': '024205c4130000e23390a7c200000001' }
REQUEST reading response's body
REQUEST finish init function https://www.mabbank.com/
REQUEST response end https://www.mabbank.com/ 503 { date: 'Wed, 22 Apr 2020 05:46:48 GMT',
  'content-type': 'text/html; charset=UTF-8',
  'transfer-encoding': 'chunked',
  connection: 'close',
  'x-frame-options': 'SAMEORIGIN',
  'set-cookie':
   [ '__cfduid=dcb46a322884336e07ec1218bb44d86821587534408; expires=Fri, 22-May-20 05:46:48 GMT; path=/; domain=.mabbank.com; HttpOnly; SameSite=Lax' ],
  'cache-control': 'no-cache',
  'expect-ct':
   'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  vary: 'Accept-Encoding',
  'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
  'x-content-type-options': 'nosniff',
  server: 'cloudflare',
  'cf-ray': '587d0be68ee7e233-SIN',
  'cf-request-id': '024205c4130000e23390a7c200000001' }
REQUEST end event https://www.mabbank.com/
REQUEST has body https://www.mabbank.com/ 11151
REQUEST emitting complete https://www.mabbank.com/
{ CloudflareError: 1020, Access Denied (Custom Firewall Rules)
    at validateResponse (/opt/app/node_modules/cloudscraper/index.js:281:11)
    at onCloudflareResponse (/opt/app/node_modules/cloudscraper/index.js:222:5)
    at onRequestResponse (/opt/app/node_modules/cloudscraper/index.js:205:5)
    at Request.<anonymous> (/opt/app/node_modules/cloudscraper/index.js:149:7)
    at Object.onceWrapper (events.js:286:20)
    at Request.emit (events.js:198:13)
    at Request.EventEmitter.emit (domain.js:448:20)
    at Request.<anonymous> (/opt/app/node_modules/request/request.js:1154:10)
    at Request.emit (events.js:198:13)
    at Request.EventEmitter.emit (domain.js:448:20)
    at IncomingMessage.<anonymous> (/opt/app/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:286:20)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1143:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'CloudflareError',
  message: '1020, Access Denied (Custom Firewall Rules)' }

Green-Cat commented 4 years ago

Same issue here, on multiple sites. Works in browser, but not through cloudscraper Changing the TLS ciphers as suggested in other issues seems to have no effect.

happyday517 commented 4 years ago

Same issue too.

poynt2005 commented 4 years ago

Same issue here...

andress134 commented 4 years ago

CloudflareError: 1020, Access Denied (Custom Firewall Rules) at validateResponse (/root/js/cloudscraper/index.js:281:11) at onCloudflareResponse (/root/js/cloudscraper/index.js:222:5) at onRequestResponse (/root/js/cloudscraper/index.js:205:5) at Request.<anonymous> (/root/js/cloudscraper/index.js:149:7) at Object.onceWrapper (events.js:422:26) at Request.emit (events.js:315:20) at Request.<anonymous> (/root/js/node_modules/request/request.js:1161:10) at Request.emit (events.js:315:20) at IncomingMessage.<anonymous> (/root/js/node_modules/request/request.js:1083:12)

same problem. Yesterday I saw on the cloudflare website that it said "update"

buihoangvu commented 4 years ago

Same issue tooo...

Dima22 commented 4 years ago

Have the same problem. Sometimes also have next: { ParserError:

Cloudflare may have changed their technique, or there may be a bug.

Bug Reports: https://github.com/codemanki/cloudscraper/issues

Check the detailed exception message that follows for the cause.

challengeId (jschl_vc) extraction failed at onChallenge (/var/www/bt/node_modules/cloudscraper/index.js:317:21) at onCloudflareResponse (/var/www/bt/node_modules/cloudscraper/index.js:247:12) ...

Thats because script trying to find match = body.match(/name="jschl_vc" value="(\w+)"/); But if I look at html code I see <input type="hidden" value="6ea98459f78d801733a08c06b2a090ca" id="jschl-vc" name="jschl_vc"/> Field VALUE located before NAME and ID between them

Tobbe commented 4 years ago

Other cloudflare scrapers have this problem too https://github.com/VeNoMouS/cloudscraper/issues/185 https://github.com/VeNoMouS/cloudscraper/issues/197

bestplay9384 commented 4 years ago

Guys, can you check if this fix works for you?

https://github.com/bestplay9384/cloudscraper/commit/a8d45e97a81b856160f94e1d96114d4498e2fffb

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

happyday517 commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

yes, this works for me, thanks

stanleyxu2005 commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

@bestplay9384 Great! Your patch works well for me as well.

The original code looks a bit unstable to extract values from node. I appreciate the way you are doing in. An ultimate solution is probably using a DOM parser like cheerio to find and extract value from a node.

andress134 commented 4 years ago

We should think about replacing the cloudscraper, it seems to be dead, no one has dealt with updates for 2 months, although several errors have been reported

Dima22 commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

Don't work for me. I haven't setTimeout(function(){ ... },4000); Haven't all challenge code. Instead I have the new script:

window._cf_chl_opt={
      cvId: "1",
      cType: "non-interactive",
      cNounce: "55205",
      cRay: "5885476c2b20f981",
      cHash: "b6325db0634ad7e",
      cRq: {
        d: "i/e6L9mNU0Wx4EEDXDKILOHqqIeQZHqdTnud0VUS6oEak4TnVqioSS2PKNOBpUnvBkcUVKuYNRaOX1uth0J77BagcXJfekfI/1DJiu6tSQvzkIR1ZG9XPLFAdDhcVkFnrtkNd+ynKoNcCHGB6QccWxynrymxFijfoT6bo4oULfP8G0MH5OsGWzAMMs3/IZs6VTA1q9GtKyv3ZvAe+dlV6GLR+50miyDsoKIZeLsAIH9HjhS7fGRegId/PyOK15LjiQdhg1yTDkfl3J8cumCgiFCan/p25GhmJUkh3985yB74FEGzDOTlUR3Sm5MqLwBL",
        t: "MTU4NzYyMDczMi44MjkwMDA=",
        m: "PaC2ctNpbo+tHnNEsBhyftv81KgQpM6Pi/M7oPDDDEM=",
        i1: "SOpF4po/BqGPmVAiq7vIqw==",
        i2: "2MtEnwHnzWOUFN04e//L3w==",
      }
    }
    window._cf_chl_enter = function(){window._cf_chl_opt.p=1};

    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
    b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
    b(function(){
      var a = document.getElementById('cf-content');a.style.display = 'block';
      var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
      var trkjs = isIE ? new Image() : document.createElement('img');
      trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=5885476c2b20f981");
      trkjs.id = "trk_jschal_js";
      trkjs.setAttribute("alt", "");
      document.body.appendChild(trkjs);

      document.body.appendChild(trkjs);
      var cpo=document.createElement('script');
      cpo.type='text/javascript';
      cpo.src="/cdn-cgi/challenge-platform/orchestrate/jsch/v1";
      cpo.onload=function(){window._cf_chl_enter()};
      document.head.appendChild(cpo);

    }, false);
bestplay9384 commented 4 years ago

@Dima22 can you provide an URL you was requesting? Or at last the whole HTML of the website?

Dima22 commented 4 years ago

@bestplay9384 https://betscsgo.cc/ Maybe this script runs just once for IP. Because now in my browser I have challenge code. Or maybe it runs periodically

bestplay9384 commented 4 years ago

@Dima22 yea it looks like that it is something new CF provided. I was finally able to see this protection while hard-refreshing view-source:https://betscsgo.cc/ in incognito mode multiple times :) I will try to come up with a way of bypassing this, however as far as i can see eg. here: https://github.com/VeNoMouS/cloudscraper/commit/72f2204b36532242f7cf65510f4bb5602cf8ed21, others are just detecting this "new way". I'll give it a try if i find some time!

ChesterDenn commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

This work for me also... thanks!

bestplay9384 commented 4 years ago

Guys, can you check if this fix works for you? bestplay9384@a8d45e9 For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

@bestplay9384 Great! Your patch works well for me as well.

The original code looks a bit unstable to extract values from node. I appreciate the way you are doing in. An ultimate solution is probably using a DOM parser like cheerio to find and extract value from a node.

@stanleyxu2005 i have found some time gap - all the stuff with DOM parsing are now using cheerio as you suggested. You can take a look on my forked repo :) I think there is still something missing in this bypass - probably no new hCaptcha detection & it is still not detecting the new challengeUI @Dima22 provided (i think they are just testing it randomly, because i can get it in 1 per 30 requests...) and it is still not able to bypass the new UI :)

andress134 commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

same error using this new version, 2 times working and now again ask for captcha

if I try 4 times in a row, it works 2 times, and then it asks for captcha

url tested: https://cyberwarblog.xyz/

// on this url also dont work https://fatality.win/

dr-nyt commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

Thanks a lot, works like a charm. If this repo drops would you consider picking it up? I really don't want to use Selenium etc ;(

andress134 commented 4 years ago

Guys, can you check if this fix works for you? bestplay9384@a8d45e9 For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

Thanks a lot, works like a charm. If this repo drops would you consider picking it up? I really don't want to use Selenium etc ;(

Just work some times :D if u are testing many times in same url will get same error

for me work 2 times, and if i try again in same url i get captcha error

for ex i have tested on https://cyberwarblog.xyz/ https://fatality.win/

I try 10 times, 4 times it works, and 6 times I get a captcha error in the same url, using different servers, without using proxies

Zirpoo commented 4 years ago

It may not be the best way to do it, but it worked for me.

` const data = {}; let inputs = body.match(/(<input(.*)\/>)/g);

for (const input of inputs) {
    const inputName = input.match(/(name=\"\w+\")/)[0].slice(6, -1);

    switch (inputName) {
        case "r":
        case "jschl_vc":
        case "pass":
            const inputValue = input
                .match(/(value=\"(.[^"]+)\")/)[0]
                .slice(7, -1);

            data[inputName] = inputValue;
            break;
    }
}

`

minas90 commented 4 years ago

Hi guys, Has anyone solved this problem? Cloudscraper is no longer maintained?

sayem314 commented 4 years ago

Shit. After 3 days of debugging and strained eyes, I have finally cracked it. Cloudscraper method of cracking the answer is not really efficient or accurate. I will post a new repo soon, already written most part and it's based on got.

sayem314 commented 4 years ago

Added repository @sayem314/hooman

TODO

image

Test are done on https://cf-js-challenge.sayem.eu.org

francisvalois commented 4 years ago

@sayem314 This will only works for the old challenge (who only do jsFuck and basic DOM validation) but not for the new challenge they added in april. I don't know if they started to release it worldwide yet.

Also, i don't think it's a good idea to run arbitrary code in Eval. A malicious server could send any JS code and take control of the process/server. You should at minimum use a sandbox and parser to limit or remove malicious code.

sayem314 commented 4 years ago

@francisvalois I see. Then probably those challenges are not available for me yet. And about eval yes it should be run inside sandbox but again hooman is meant for accessing known sites but still I will try to safe it on next version :) Thanks.

If anyone has new challenge HTML from browser could you please share here? Please upload to https://paste.ubuntu.com/

Edit: code are now run safely using vm2. Commit https://github.com/sayem314/hooman/commit/7332cc2d40c0be3608e6a91718831f5ed2f6f896

btpython commented 4 years ago

@sayem314 How do you implement that in cloudscraper ?

btpython commented 4 years ago

@sayem314 your script does not work with yggtorrent it leave me with a 403 response code (Forbidden)

sayem314 commented 4 years ago

@btpython 403 === captcha error

narra-dev commented 4 years ago

Guys, can you check if this fix works for you?

bestplay9384@a8d45e9

For my case it is just right but i did fix only the casual JS challenge stuff, didn't test any captchas etc. Tested on https://www.apotea.se/

Nice fix, thanks. The only problem I see is that console.log(stringBody); line keeps terminal window spammed with unnecessary html