thibauts / node-google-search-scraper

Google search scraper with captcha solving support
MIT License
89 stars 40 forks source link

Added the possibility of getting multiple results #7

Open tommelo opened 7 years ago

tommelo commented 7 years ago

As mentioned in the Issue #6 I have added the possibility of getting multiple results:

var scraper = require('google-search-scraper');

var options = {
  query: 'nodejs',
  limit: 10,
  fullResult: true
};

scraper.search(options, function(err, results) {
  if(err) throw err;
  // Array of url's
  console.log(results);
})
steventsao commented 6 years ago

Thanks @tommelo . This prevents an issue where a crawler gets stuck when there are fewer results than the set limit. The original result callback doesn't inform the caller when there are no more urls, and makes the following implementation fragile:

let urls = [];
 scraper.search(options, (err, url) => {
      urls.push(url);
      // handleResults is never called if there are fewer results.
      if (urls.length === options.limit) {
        handleResults(name, urls);
      }
    });