Data pipeline: review data missing. scraper ended successfully, but processed review count << local review count

[rivernews]

Slack message of the scraper log.

Org: groupon

Hint: observe the url page number & the processed review count. You can see basically one page contains 10 reviews.

[rivernews]

How to cope with this?

• Just re-scrape again. But we have to be aware that our current mechanism will halt upon duplicated review id. We may:
  • [x] (required) Delete the entire data for groupon in S3, then redcrape again.
    • If we're storing collided reviews as well, then it's better to start over especially for debug purpose, so we can observe duplication particularly for that scraping session.
  • [x] (required) OR, Remove the duplication prevention mechanism, [but add some sort of collision handling], then try again. How to we do collision handling?
    • [x] I'll recommend you add timestamp suffix to all reviews. AND, whenever a collision occurs, name the review file as collision.<review id>.<timestamp>.json. This is so that it's easier to search by prefix, which is the only native way to search that S3 provides.
    • Make sure this filename change also has to be compatible with existing data.
• Add more log:
  • about "no next button", or about "no next pagination":
    • [ ] (required) When local review count - processed review count >= 3% of local review count, then upload the current page html content to s3, send an email to raise attention, and then include the s3 link of webpage.
  • [ ] (required) About duplicated review found. Actually we want to raise attention to these duplicated reviews so we may just email. But we don't want to have our inbox flood with emails, so we may just generate one email per scraper session. So we might record the duplicated review object globally within our scraper session, and output them at the end of task. Better if this email can also be sent upon exception.

But why?

Eventually we want to find out why we lose so much reviews. Or why the scraper did not see the next page link and exit by 0?

[rivernews]

Investigation

When we re-tried groupon, we got around 6xx results. Seems like the abortion is due to no studout in 10 minutes. This is due to we tweak the loglevel in travis to 2, warning. Changing to 3 INFO should solve this issue. The original problem's cause is still not identified.

After we run the 2nd time, it's now:

Processed reviews count: 2443/2696
Duration: 0h:32min:22s.905

Seems like not a big deal here looking at the review count rate. However we do want to verify if there is no further page.

Based on the page number, we believe that we did retrieve all we have. So this indicate that there's a gap between the shown local count and the actual reviews available.