maxCopell / tripadvisor-scraper

Scrape Tripadvisor restaurant, hotels, and places.
https://apify.com/maxcopell/tripadvisor
47 stars 29 forks source link

Only able to get 20 reviews #58

Closed LouisChoiniere closed 3 years ago

LouisChoiniere commented 3 years ago

When scraping the data of a restaurant I'm only able to get 20 reviews

When looking through the code I found that it gets reviews 20 at the time with a while true I belive there is a issue with the exit condition where it always exit on the first iteration of the loop

https://github.com/maxCopell/tripadvisor-scraper/blob/master/src/tools/general.js

 while (true) {

    //...
    //...

    if (reviews.length < limit || result.length >= maxReviews || shouldSlice) break;
}

My input

{
  "locationId": "4879161",
  "includeRestaurants": true,
  "includeAttractions": false,
  "includeHotels": false,
  "includeReviews": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxReviews": 0,
  "maxItems": 1,
  "language": "en",
  "currency": "CAD",
  "debugLog": true,
  "checkInDate": "",
  "includeTags": false
}
LouisChoiniere commented 3 years ago

When maxReviews is 0 in the input shouldSlice will be false but this will mean result.length >= maxReviews will always be true therefore it will always exit on the first iteration

FayzulSaimun commented 3 years ago

Hey, I am also getting 20 reviews per hotel. It is not scraping all reviews. What is the solution for this? My input was this:

{ "locationFullName": "Cancun, Mexico", "maxItems": 0, "includeRestaurants": false, "includeHotels": true, "includeAttractions": false, "includeTags": false, "includeReviews": true, "maxReviews": 0, "language": "en", "currency": "USD", "proxyConfiguration": { "useApifyProxy": true }, "debugLog": true }

metalwarrior665 commented 3 years ago

We will fix this very soon (next week latest). @zpelechova Please mark this for next week

FayzulSaimun commented 3 years ago

@metalwarrior665 there is also a problem with single hotel/restaurants scraping. The actor scrape other hotels/restaurants details also.

LouisChoiniere commented 3 years ago

Fixed on Apify using input from original issue