search

zlurad / serp-parser

Nodejs lib to parse Google SERP html pages
MIT License
43 stars 12 forks source link

Parsing doesn't work anymore #92

Open marcoschwartz opened 3 years ago

marcoschwartz commented 3 years ago

Hello,

We have been using your package in production for over 3 months now, worked well but a few days ago it started to not work anymore with the error:

{ currentPage: 1, keyword: '', organic: [], pagination: [], relatedKeywords: [], error: 'Not Google nojs page' }

We tried on local machines to check if it was not Google blocking us - but same result. HTML is returned fine - but it just doesn't get parsed properly. Any ideas?

badluck13 commented 3 years ago

Hi @marcoschwartz are you sure you are parsing the google no-js page?

Maybe Google did an update on nojs results HTML, we will check it later, just wanted to hint you the possible issue, and if Google did change HTML it can be few weeks until this is resolved, you can help by submitting PR

Thanks

marcoschwartz commented 3 years ago

Hi,

I confirm, I tried with the example you are giving (with request), as well as with axios - same result. I am not sure how it can be solved - but if it can help we switched to using this package: https://www.npmjs.com/package/serp

However it is more like a 'black box' compared to your package, that's why we prefer yours :)

skwasha commented 2 years ago

I can confirm this as well (not working with the no-js results page).

skwasha commented 2 years ago

@badluck13 FWIW - I came back to check on this. Since it hadn't been updated I did a tiny bit of digging. It appears the issue is caused here:

    if ($('body').attr('jsmodel') === 'TvHxbe') {
      this.parseGoogle(opt);
    } else {
      this.serp.error = 'Not Google nojs page';
      return;
    }
  }

If the jsmode is changed to hspDDf then all is well.

I didn't do much more checking - so, I don't know if just changing that would be a good fix or if it'd just be a band-aid.