godfriedmeesters / scraper

As part of DiffScraper, one or more bots can be deployed. Ready-to-use bots are provided that can extract offers from mobile applications, mobile websites and desktop websites.
GNU General Public License v3.0
2 stars 0 forks source link

more proxies #9

Closed godfriedmeesters closed 3 years ago

godfriedmeesters commented 3 years ago

Noted a ban by Opodo web ( see _id with value HR-KiXgBXZp0CmfYs-ov in Kibana).

Would be nice to have more proxies

godfriedmeesters commented 3 years ago

Now there are two proxies

godfriedmeesters commented 3 years ago

Seems working fine with two proxies in proxies.json and code like this:

const inputData = JSON.parse(
            fs.readFileSync("proxies.json")
        );
        //
        var use_proxy = Math.random() < 0.6;
        //var use_proxy = false;
        const proxies = inputData.proxies;

        const proxy_index = Math.floor(Math.random() * proxies.length);

        if (use_proxy) {
            job.data.params.proxy = proxies[proxy_index];
            logger.info("Selected proxy " + proxies[proxy_index]);
        }
        else { logger.info("No proxy selected") }
bkrumnow commented 3 years ago

I think we safely close this one in the favour of #16