codemanki / cloudscraper

--DEPRECATED -- 🛑 🛑 Node.js library to bypass cloudflare's anti-ddos page
MIT License
599 stars 139 forks source link

Putting URLSearchParams into request #322

Closed MrKafr closed 4 years ago

MrKafr commented 4 years ago

hello, i am trying to scrap the website where the Cloudflare is inplemented. Before the implementation i called it like this

     const url = "website";
     form = new URLSearchParams(`items=${itemsId}&scrap=${itemsValue}&for_user=0&csrf=${csrf}`);
     return fetch(url, {
          method: "POST",
          body: form,
          headers: {
               cookie: config.cookies
          }
     })
          .catch(error => { errorHandling(error) })
          .then(res => res.json())
          .then(body => {
     })
}

And this works. Now, after the implementation of Cloudflare i try to use this

var options = {
          uri: "website",
          formData: { items: itemsId, scrap: itemsValue, for_user: 0, csrf: csrf },
          headers: { cookie: config.cookies }
     };
     cloudscraper.post(options)
          .catch(error => { errorHandling(error) })
          .then(body => {
          })
}

but it looks like the itemsId is parsed wrongly and it doesnt work. When i do the console.log of the formdata cloudscraper solution i get this

formData:
   { items: '%5B%228483450990%22%5D',
     scrap: 557,
     for_user: 0,
     csrf:
      'csfr' },

the webside needs to get data like this. items=%5B%228483450990%22%5D&scrap=557&for_user=0&csrf=csfr

Screen https://imgur.com/a/KtVaJ5o
MrKafr commented 4 years ago

I found the issue https://github.com/codemanki/cloudscraper/issues/319 Iam thinking if this isnt a same problem... maybe the formData parse it with wrong &... Can i somehow console.log raw request data the cloudscraper send?

KostyaChoporov commented 4 years ago

Hotfixed with just replace each & to & in index.js:376

MrKafr commented 4 years ago

@KostyaChoporov

Hotfixed with just replace each & to & in index.js:376

Can you please give me more information how to fix it?

EDIT: i did some testing and it looks like this onChallenge function is never called in my solution. But the form data is still sending wrongly

EDIT2: Okay, it looks like the website is turning its cloudflare protection on and off every few weeks (my solution with fetch is now working), but cloudscraper should work even if there is no cloudflare protection, right? I would like to solve this problem before the website turn it on again,. Now i can provide working information from fetch and broken ones from cloudscraper at once.

codemanki commented 4 years ago

Should be fixed in 4.6.0. Please update and try again