search

facundoolano / app-store-scraper

scrape data from the itunes app store
MIT License
1.01k stars 240 forks source link

Check if reviews is a valid type array #18

Closed bryan closed 7 years ago

bryan commented 7 years ago

If results.feed.entry exists but is not a valid type array (i.e. dictionary) in our case, this error occurs. The resulting json file returns a dictionary for this entry when there are no reviews in that page.

TypeError: reviews.slice is not a function
    at cleanList (.../node_modules/app-store-scraper/lib/reviews.js:10:18)
    at process._tickDomainCallback (internal/process/next_tick.js:129:7)

A simple check to see if it's a valid array would be sufficient for a fix in this case.

facundoolano commented 7 years ago

This has been brought up before and fixed, what version of the module are you using?

bryan commented 7 years ago

Currently, app-store-scraper: 0.5.1 obtained via NPM

screen shot 2017-03-05 at 10 30 42 am
facundoolano commented 7 years ago

Can you show me an example to reproduce the error?

bryan commented 7 years ago

Take a look at the entry section which is being sliced between these two json responses: https://itunes.apple.com/us/rss/customerreviews/page=1/id=389801252/sortby=mostRecent/json

versus

https://itunes.apple.com/ro/rss/customerreviews/page=1/id=389801252/sortby=mostRecent/json

a b

facundoolano commented 7 years ago

I mean a code snippet with the data you need to input to generate the error you are seeing.

bryan commented 7 years ago

I have a bit of things on my hands currently and won't be able to create a test example at the moment. You can feel free to test it on your own. I temporarily fixed things locally on my end, but the error derives from results.feed.entry being a Dictionary instead of an Array of Dictionary.

facundoolano commented 7 years ago

I was asking for something as simple as this:

store = require('app-store-scraper');
store.reviews({id: 389801252});

But well, now that I can reproduce I can look into fixing it. I also have a lot of things in my hands, so I can't tell when that will happen.

bryan commented 7 years ago

I had to go through the different locales before the error popped up for that one instance - hence why I gave you the JSON format and specific examples to my issue and can't copy and paste my implementation without having to refactor it to run properly, sorry about that inconvenience.

You can use the locales here and loop through all the pages 1 to 10 for each locale, sort by RECENT, and the id is correctly listed as 389801252 for Instagram for which I ran into the bug: 

var regionList = ["ae", "ag", "ai", "al", "am", "ao", "ar", "at", "au", "az", "bb", "be", "bf", "bg", "bh", "bj", "bm", "bn", "bo", "br", "bs", "bt", "bw", "by", "bz", "ca", "cg", "ch", "cl", "cn", "co", "cr", "cv", "cy", "cz", "de", "dk", "dm", "do", "dz", "ec", "ee", "eg", "es", "fi", "fj", "fm", "fr", "gb", "gd", "gh", "gm", "gr", "gt", "gw", "gy", "hk", "hn", "hr", "hu", "id", "ie", "il", "in", "is", "it", "jm", "jo", "jp", "ke", "kg", "kh", "kn", "kr", "kw", "ky", "kz", "la", "lb", "lc", "lk", "lr", "lt", "lu", "lv", "md", "mg", "mk", "ml", "mn", "mo", "mr", "ms", "mt", "mu", "mw", "mx", "my", "mz", "na", "ne", "ng", "ni", "nl", "no", "np", "nz", "om", "pa", "pe", "pg", "ph", "pk", "pl", "pt", "pw", "py", "qa", "ro", "ru", "sa", "sb", "sc", "se", "sg", "si", "sk", "sl", "sn", "sr", "st", "sv", "sz", "tc", "td", "th", "tj", "tm", "tn", "tr", "tt", "tw", "tz", "ua", "ug", "us", "uy", "uz", "vc", "ve", "vg", "vn", "ye", "za", "zw"];

I'm going to close this, feel free to reopen this or create a new issue when you decide to update it. Thanks.

facundoolano commented 7 years ago

I'm leaving this open since I can reproduce the error with id 389801252.

facundoolano commented 7 years ago

this was fixed by the changes in #19