NikolaiT / se-scraper

Javascript scraping module based on puppeteer for many different search engines...
https://scrapeulous.com/
Apache License 2.0
543 stars 123 forks source link

Set different geolocation for every request #36

Open mpassion opened 5 years ago

mpassion commented 5 years ago

I need to set different gelocation for every search query to Google. I set manually permissions in Browser.js (puppeteer-cluster catalog) because in se-scraper I don’t have access to browser object (or maybe I’m wrong?)

context = yield chrome.createIncognitoBrowserContext(); 
context.clearPermissionOverrides();  
context.overridePermissions('https://www.google.pl', ['geolocation']);

Next I overwritten GoogleScraper due to extends search_keyword method with geolocation params:

async search_keyword(keyword) {
    await this.page.setGeolocation({
               latitude: latVar,
               longitude: longVar
    });
    const input = await this.page.$('input[name="q"]');
    await this.set_input_value(`input[name="q"]`, keyword);
    await this.sleep(50);
    await input.focus();
    await this.page.keyboard.press("Enter");
}

I noticed that it’s necessary to reload the page to update Google results with new geo. When I put this code after Enter in search_keyword

await this.page.evaluate(() => {
    location.reload(true)
});

I got: Error: Execution context was destroyed, most likely because of a navigation

I have also tried with put

await input.focus();
await this.page.keyboard.press("Enter");

but the same as above.

How to properly refresh the page or how to get the results related with geo params?

c-o-x commented 5 years ago

Same issue as above.

NikolaiT commented 5 years ago

Honestly I don't know what the solution is for this. I don't have currently the time to debug this, can anyone step in?