Customization features

[pim97]

Hi, thanks for creating such a wonderful module

I am wondering if it's possible to add the following features:

Setting your own user-agent Setting your own proxy Receiving the cookies in a json format instead of full HTML. That's what also #8 meant I think

I'll also be looking if I can contribute to this repo

[JimmyLaurent]

Good news, everything is already implemented. One note though, just be carefull when you're changing the user-agent, it may affect the bypass and be less effective.

const cloudflareScraper = require('cloudflare-scraper');

(async () => {
 try { 
    const jar = cloudflareScraper.jar();
    const url = 'https://your-url.com'; 
    const response = await cloudflareScraper.get(url, { 
       jar,
      headers: {
         'User-Agent': 'your user agent'
      },
      proxy: 'http://localhost:8080' // Here you can pass a http proxy, or you can use HTTP_PROXY and HTTPS_PROXY environment variables
    });
    const cookiesObject = jar.getCookies(url); // Here is your cookie object
    const cookieJson = JSON.stringify(cookieObject, null, 2); // Here is a json string of the cookie object
    const cookieStr = jar.getCookieString(url); // Here is a cookie string
    console.log(cookieStr); 
} catch (error) {
   console.log(error); 
}
})();

[pim97]

Good news, everything is already implemented. One note though, just be carefull when you're changing the user-agent, it may affect the bypass and be less effective.

const cloudflareScraper = require('cloudflare-scraper');

(async () => {
 try { 
    const jar = cloudflareScraper.jar();
    const url = 'https://your-url.com'; 
    const response = await cloudflareScraper.get(url, { 
       jar,
      headers: {
         'User-Agent': 'your user agent'
      },
      proxy: 'http://localhost:8080' // Here you can pass a http proxy, or you can use HTTP_PROXY and HTTPS_PROXY environment variables
    });
    const cookiesObject = jar.getCookies(url); // Here is your cookie object
    const cookieJson = JSON.stringify(cookieObject, null, 2); // Here is a json string of the cookie object
    const cookieStr = jar.getCookieString(url); // Here is a cookie string
    console.log(cookieStr); 
} catch (error) {
   console.log(error); 
}
})();

Could this be added to the readme? is something more people are looking for. Thanks btw for the quick reply.

[JimmyLaurent]

This library is young, It still was a POC one week ago. Until now I was more focused on the code than writing some documentation. It's almost stable now, I'll add some notes in the readme.

Note: for more informations, you can check this documentation.