perBrowserOptions - Proxy Question?

lazybotter commented 4 years ago

Hi,

I am trying to create an application that will run multiple instances of puppeteer, each instance will have its own proxy. The new "perBrowserOptions" is good but how can I set the proxy based on the Taskfunction that has been queued when using "CONCURRENCY_BROWSER"?

Here is an example to try and explain what i mean...

`const {Cluster} = require('puppeteer-cluster');

(async ()=>{

const proxies = [];

//Create the cluster
const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_BROWSER,
    maxConcurrency: 4,
    // puppeteerOptions : {
    //     headless : false,
    //     args : [
    //         '--no-sandbox',
    //         '--disable-setuid-sandbox',
    //         '--disable-infobars',
    //         '--window-position=0,0',
    //         '--ignore-certifcate-errors',
    //         '--ignore-certifcate-errors-spki-list',
    //         `--proxy-server=${proxies.shift()}`
    //     ]
    // },
    //Something like this would be nice
    browserLaunchedOption:()=>{
        //return options for browser
        return {
            headless : false,
            args : [
                '--no-sandbox',
                '--disable-setuid-sandbox',
                '--disable-infobars',
                '--window-position=0,0',
                '--ignore-certifcate-errors',
                '--ignore-certifcate-errors-spki-list',
                `--proxy-server=${proxies.shift()}`
            ]
        }
    }
});

const loginFBWithProxyTask = async ({page,data})=> {

    proxies.push(data.proxy.ip);

    //Somehow need to set the proxy here... from data.proxy.ip
    //Proxy must match account

    await page.authenticate({
        username: data.proxy.user,
        password: data.proxy.pass
    });

    await page.goto("http://facebook.com", {waitUntil: "domcontentloaded"});
    //await page.type()
    //perform login etc....
};

//check DB every 30 sec
setInterval(function () {

    //fetch some accounts from a db

    const accounts = [  //returned from db just for example
        {
            fb_username : "Joe Blogs",
            fb_password : "pass",
            proxy : {
                ip : "51.91.195.74:29842",
                user : "proxyuser",
                pass : "password"
            }
        },
        {
            fb_username : "Jame Smith",
            fb_password : "pass",
            proxy : {
                ip : "21.21.195.24:29842",
                user : "myproxyuser",
                pass : "myproxypass"
            }
        },
        {
            fb_username : "Mary Poppins",
            fb_password : "pass",
            proxy : {
                ip : "21.21.295.24:29842",
                user : "myproxyuser",
                pass : "myproxypass"
            }
        }];

    //QUEUE UP Tasks to perform perBrowser
    for (const account in accounts) {

        cluster.queue(account,loginFBWithProxyTask);
    }

},30000);

})();`

I just can't work out how i can set the proxy per browser launch to match the account that is fetched/queued...

Hope some of that made sense, thanks!

thomasdondorf commented 4 years ago

You mean something like: cluster.queueForWorker(0, data).

This sounds like an interesting feature. But it's currently not supported.

lazybotter commented 4 years ago

Im not sure, would something like cluster.queueWithBrowserOptions(data,taskFunction,browserOptions) work?

Or add another prop to cluster.queue(data,taskFunction,browserOptions) then if "CONCURRENCY_BROWSER" is used, "pupeteerOptions" can be defined for each task in the queue.

Thanks