thibauts / node-google-search-scraper

Google search scraper with captcha solving support
MIT License
89 stars 37 forks source link

Exception met with certain queries #13

Closed jesuscc1993 closed 2 years ago

jesuscc1993 commented 6 years ago

When sending certain queries the following exception is thrown:

TypeError: Parameter "url" must be a string, not undefined
    at Url.parse (url.js:88:11)
    at Object.urlParse [as parse] (url.js:82:5)
    at .<anonymous> ({removed}\node_modules\google-search-scraper\index.js:102:27)
    at exports.each ({removed}\node_modules\cheerio\lib\api\traversing.js:226:24)
    at extractResults ({removed}\node_modules\google-search-scraper\index.js:95:21)
    at onPage ({removed}\node_modules\google-search-scraper\index.js:32:26)
    at Request._callback ({removed}\node_modules\google-search-scraper\index.js:86:9)
    at Request.self.callback ({removed}\node_modules\request\request.js:122:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)

Example query: { query: 'ca', limit: 1 }

PauloPatoleia commented 6 years ago

I'm having the same problem

rohanleach4 commented 5 years ago

There seems to be an issue if the first response is undefined. I fixed it by wrapping a condition around it to prevent this error. I am short on time at the moment, but I would love to explore why this is happening. Until then, here is a fix: go to:

\node_modules\google-search-scraper\index.js:102:27

Replace:

    var parsedUrl = url.parse(elemUrl.attr("href"), true);
    if (parsedUrl.pathname === '/url') {
      item['url'] = parsedUrl.query.q;
    }

with:

  if(elemUrl.attr("href")){
    var parsedUrl = url.parse(elemUrl.attr("href"), true);
    if (parsedUrl.pathname === '/url') {
      item['url'] = parsedUrl.query.q;
    }
  }
jesuscc1993 commented 2 years ago

Closing issue due to inactivity.