update scraper dependencies

facundoolano / aso

Tools for app store optimization on iTunes and Google Play

MIT License

720 stars 144 forks source link

update scraper dependencies #19

Closed facundoolano closed 5 years ago

facundoolano commented 5 years ago

First stab at bumping the app-store-scraper and google-play-scraper dependencies. We'll need to test each method to make sure it's working (this is what you get for not having tests :trollface:).

Help testing the different methods is welcome!

[x] itunes.app
[x] gplay.app
[x] itunes.scores
[x] gplay.scores
[x] itunes.visibility
[x] gplay.visibility
[x] itunes.suggest.CATEGORY
[x] itunes.suggest.SIMILAR
[x] itunes.suggest.COMPETITION
[x] itunes.suggest.ARBITRARY
[x] itunes.suggest.KEYWORDS
[x] itunes.suggest.SEARCH
[x] gplay.suggest.CATEGORY
[x] gplay.suggest.SIMILAR
[x] gplay.suggest.COMPETITION
[x] gplay.suggest.ARBITRARY
[x] gplay.suggest.KEYWORDS
[x] gplay.suggest.SEARCH

facundoolano commented 5 years ago

~BUG: gplay.scores broken because the updated field is now an integer in gplay (while it's still a timestamp string in itunes)~ Fixed.

facundoolano commented 5 years ago

(potential) BUG: seeing some ETIMEDOUT errors for itunes.scores

facundoolano commented 5 years ago

BUG: the updated timestamp from google play currently reflects the release date.

https://github.com/facundoolano/google-play-scraper/issues/212

facundoolano commented 5 years ago

~BUG (app-store-scraper)~ Fixed.

> itunes.visibility(284882215).then(console.log)
> (node:18883) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'split' of undefined
    at cleanApp (/Users/facundo/dev/facundoolano/aso/node_modules/app-store-scraper/lib/list.js:11:67)
    at Array.map (<anonymous>)
    at /Users/facundo/dev/facundoolano/aso/node_modules/app-store-scraper/lib/list.js:42:17
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:118:7)
(node:18883) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)

facundoolano commented 5 years ago

~BUG~ Fixed.

itunes.suggest({strategy: itunes.CATEGORY, appId: 'com.facebook.Facebook', num: 5}).then(console.log, console.log)
  app-store-scraper Making request: https://itunes.apple.com/lookup?id=284882215&country=us&entity=software {} undefined +0ms
>   app-store-scraper Finished request +94ms
  app-store-scraper Making request: http://ax.itunes.apple.com/WebObjects/MZStoreServices.woa/ws/RSS/topfreeapplications//genre=6005/limit=100/json {} undefined +1ms
  app-store-scraper Finished request +267ms
  app-store-scraper Making request: https://itunes.apple.com/lookup?id=,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,&country=us&entity=software {} undefined +4ms
  app-store-scraper Finished request +637ms
[]

facundoolano commented 5 years ago

BUG Also caused by using a numerical id instead of the bundle id

> itunes.suggest({strategy: itunes.SIMILAR, appId: '284882215', num: 5}).then(console.log, console.log)
  aso Calling app-store-scraper +0ms similar {"country":"us","throttle":20,"fullDetail":true,"id":"284882215"}
>   app-store-scraper Making request: https://itunes.apple.com/us/app/app/id284882215 {"X-Apple-Store-Front":"143441,32"} undefined +4ms
  app-store-scraper Finished request +140ms
[]