facundoolano / google-play-scraper

Node.js scraper to get data from Google Play
MIT License
2.28k stars 624 forks source link

From 100 appIds, one consistently returns no result #400

Closed Giszmo closed 4 years ago

Giszmo commented 4 years ago

I'm monitoring about 100 apps and it worked great until a week ago or so when one app started returning Error: App not found (404) yet when checking with a browser, the app is there just fine. What could be the reason for this discrepancy?

Edit: Just to confirm, 27 days later this one appId is still the only one not working.

walrus543 commented 4 years ago

I couldn't give a number but I do have issues too for 2 weeks.

I'm stuck on a (random?) package. Example: image (Note: this package exists on Google Play)

I have to stop the process. Several downloaded icons have a size of 0 Kb (I can't open them).

walrus543 commented 4 years ago

It happened again with only 8 packages: image

EDIT: 2 minutes later image url: 'https://play.google.com/store/apps/details?id=com.gameloft.android.ANMP.GloftAGHM&hl=en&gl=us' } { Error: Error requesting Google Play:connect ETIMEDOUT 216.58.215.46:443 at /home/arnaud/AndroidAll/Google_Play_Scraper/node_modules/google-play-scraper/lib/utils/request.js:44:19 at <anonymous> at process._tickCallback (internal/process/next_tick.js:188:7) status: undefined }

yurganov commented 4 years ago

shit happens

connect ETIMEDOUT 216.58.215.46:443

that's because of your ISP

walrus543 commented 4 years ago

shit happens

connect ETIMEDOUT 216.58.215.46:443

that's because of your ISP

Same issue with the CLI client of ProtonVPN. I was hoping it could help.

Giszmo commented 4 years ago

@Primokorn's issue is not mine. My issue is not random at all. My issue is with one app consistently since more than a month and I would appreciate if somebody could check if using google-play-scraper works for them with the appId tech.insense.sensewalet.

icarcal commented 4 years ago

@Giszmo can you provide the language and country that you are using? By default, this lib uses us as country and en as lang.

The .app method adds two parameters to the link that you provide: hl and gl.

So, if I call the method like this:

gplay.app({
  appId: 'tech.insense.sensewalet'
}).then(console.log);

The real url being called is https://play.google.com/store/apps/details?id=tech.insense.sensewalet&hl=en&gl=us

Which give me the same error: Error: App not found (404)

BUT, if I call with a different country and lang, I receive the data consistently:

gplay.app({
  lang: 'pt',
  country: 'br',
  appId: 'tech.insense.sensewalet'
}).then(console.log);

My guess is that this is happening 'cause this app is available only in some countries.

Giszmo commented 4 years ago

I'm in Chile. So they de-published the app for the US market. Makes sense. Many Bitcoin business avoid the US. Thanks for clarifying.

walrus543 commented 4 years ago

For the records: my issue is solved. I no longer have a timeout. Maybe a temporary issue from my ISP as suggested. Glad for you @Giszmo :)

walrus543 commented 4 years ago

@icarcal Can I completely remove the language &hl=en&gl=us from the url? We can access to the url without these 2 parameters: https://play.google.com/store/apps/details?id=tech.insense.sensewalet

Thanks!

icarcal commented 4 years ago

@Primokorn right now there is no way of doing that, and the docs explicitly says what lang and country it defaults to.

I don’t see the point of removing those parameters, since this allow us to know exactly from what country and lang we are getting the app data.

walrus543 commented 4 years ago

@icarcal Because we miss apps which are not available in English. I can't do a "trial and error" language operation for each "app not found". I only use this tool to download icons so I don't need to know the country.

facundoolano commented 4 years ago

That default is intentional and I prefer it to omitting language by default which would produce inconsistent behavior of the same piece of code on different locations/setups.

But I would accept a PR with an option to explicitly tell you want to omit the country, e.g. if you pass {"country": false} then omit the option and use the "machine default". But this would need to be respected in all method calls.

walrus543 commented 4 years ago

@facundoolano Thanks for your understanding. Unfortunately I have no dev skill to suggest a PR. I work on Android icon packs, I'm more a designer than a developer. Thanks anyway.

icarcal commented 4 years ago

@Primokorn @facundoolano I’ll move this discussion to another issue, as a suggested feature so we can have a better history on this, ok?