search

facundoolano / google-play-scraper

Node.js scraper to get data from Google Play
MIT License
2.32k stars 629 forks source link

Variable number of reviews returned #676

Open gianlucascoccia opened 6 months ago

gianlucascoccia commented 6 months ago

The reviews methods seems to return a different number of reviews on each call. I slightly modified the example in the documentation to reproduce the issue:

import gplay from "google-play-scraper";

gplay.reviews({
  appId: 'com.facebook.katana',
  num: 3000
}).then(function(apps){
  console.log(apps.data.length);
}, console.log);

When called multiple times the output is :

300
300
150
150
150
150
1200

This used to work in the past, getting hundreds of thousands of reviews in a single call

gianlucascoccia commented 6 months ago

In this sister project they have encountered the same issue. Here I provided a workaround around the issue that appears to be working. I would submit a PR to fix the issue but my JavaScript skills are fairly limited.

petskratt commented 6 months ago

comment edited - pagination is built-in, just implement cookieJar

Variance in response seems to come from Google's loadbalancer picking different backend that doesn't know about our nextPaginationToken and responds with error. Providing NID cookie (that stores user prefs & ad data) gives me constant results.

PR on the way, in a meanwhile you can test with my fork.