search

Akdeniz / google-play-crawler

Play with Google Play API :)
Other
565 stars 204 forks source link

Limited reviews #4

Open gazerboy opened 11 years ago

gazerboy commented 11 years ago

Hi Akdeniz, You are awesome!

I am trying to get all the reviews for an app. However, it gives me an error "Too many results requested.". Could you please tell me how could I avoid this? For example this app "de.lotum.whatsinthefoto.us" has over 200K reviews but I can get only some of them :(

TestGooglePlay.java code

ReviewResponse reviews = service.reviews("de.lotum.whatsinthefoto.us", null, 500, 20); GetReviewsResponse response = reviews.getGetResponse();

Error msg:

Exception in thread "main" com.akdeniz.googleplaycrawler.GooglePlayException: Too many results requested.*» at com.akdeniz.googleplaycrawler.GooglePlayAPI.executeHttpRequest(GooglePlayAPI.java:490) at com.akdeniz.googleplaycrawler.GooglePlayAPI.executeGet(GooglePlayAPI.java:481) at com.akdeniz.googleplaycrawler.GooglePlayAPI.executeGETRequest(GooglePlayAPI.java:385) at com.akdeniz.googleplaycrawler.GooglePlayAPI.reviews(GooglePlayAPI.java:352) at com.akdeniz.googleplaycrawler.test.TestGooglePlay.reviewsCommand(TestGooglePlay.java:79) at com.akdeniz.googleplaycrawler.test.TestGooglePlay.main(TestGooglePlay.java:62)

Akdeniz commented 11 years ago

Hi @gazerboy,

GooglePlay servers limit fetching offset to be max 499 through this protocol. So it is not possible to fetch reviews beyond that.

Google Play application on your phone works with this limitation too. You can test it.

gazerboy commented 11 years ago

Hi Akdeniz, Thanks for your reply.

I saw some online companies they provide you complete reviews for various apps. How do they get all the reviews for an app? Is there any way/api to get all the reviews for an app?

With your API what is the max number of reviews I can get? 1000?

Thanks

Akdeniz commented 11 years ago

Can you give an example for these providers?

With google-play-crawler, you can fetch only 500 reviews.

gazerboy commented 11 years ago

For example (http://www.scriptrr.com/data-scrapping-get-android-google-play-app-info/). They mentioned (offline) that they can also get all the reviews. There are some developers at Freelance and elance that promise to get this kind of services.

I think your code could get more than 500 reviews. I got almost 1000 reviews just by tweaking your code as follow:

static void reviewsCommand() throws Exception {

    for (int i = 1; i < 500; i++) {
        int start = 0;
        start = i * 10;

        ReviewResponse reviews = service.reviews(
                "de.lotum.whatsinthefoto.us", null, start, 20);
        GetReviewsResponse response = reviews.getGetResponse();

        if (response.getReviewCount() == 0) {
            System.out.println("No review found!");
        }
        System.out.println(response);

    }
}

It gets an error asap the loop reaches to 49. I got around 1000 unique reviews because with each loop iteration it gets 20 reviews. I am sure there would be a way to get all the reviews :-)

Akdeniz commented 11 years ago

The service you mentioned seems to use HTML parsing. AFAIK there is not a such an API that is provided by Google for information fetching.

The code you pasted, fetches same reviews multiple times, you can find unique reviews like this:

Set<String> commentIds = new HashSet<String>();
try {
    for (int i = 1; i < 500; i++) {
    int start = 0;
    start = i * 10;

    ReviewResponse reviews = service.reviews("de.lotum.whatsinthefoto.us", null, start, 20);
    GetReviewsResponse response = reviews.getGetResponse();

    for (Review review : response.getReviewList()) {
        commentIds.add(review.getCommentId());
    }
    }
} catch (GooglePlayException ex) {
    System.out.println("TOTAL UNIQUE COMMENTS : " + commentIds.size());
}
gazerboy commented 11 years ago

Could I get all the reviews just for all 1 star or 2 star etc.? I know, I can get all 500 and then filter them out. But is there a way to directly request google for 1 star or 2 star reviews only?

Akdeniz commented 11 years ago

Nope, there are only three types of sorting for reviews! https://github.com/Akdeniz/google-play-crawler/blob/master/src/com/akdeniz/googleplaycrawler/GooglePlayAPI.java#L77

gazerboy commented 11 years ago

Could we have this feature? Or this sorting restriction is from Google?

Akdeniz commented 11 years ago

Unfortunately, it is a Google server restriction.