search

scrapehero-code / amazon-review-scraper

A basic python 3 based web scraper for extracting reviews from Amazon. Built using Selectorlib and requests
https://www.scrapehero.com/how-to-scrape-amazon-product-reviews/
54 stars 49 forks source link

How to update the code to scrap multiple pages? #4

Closed youikim closed 3 years ago

youikim commented 3 years ago

Hi there, thank you for the code. It is really helpful.

but I would like to ask which part of the code should be updated to scrap multiple pages. It only scraps the first page even though I put multiple pages in 'urls.txt'. Is there any easy fix solution for the issue?

for example, when I put five different urls like below, it only read the first one and kept repeating the same result for five times.

https://www.amazon.com/2030-Biggest-Collide-Reshape-Everything/product-reviews/1250268176/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=1 https://www.amazon.com/2030-Biggest-Collide-Reshape-Everything/product-reviews/1250268176/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=2 https://www.amazon.com/2030-Biggest-Collide-Reshape-Everything/product-reviews/1250268176/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=3 https://www.amazon.com/2030-Biggest-Collide-Reshape-Everything/product-reviews/1250268176/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=4 https://www.amazon.com/2030-Biggest-Collide-Reshape-Everything/product-reviews/1250268176/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=5

Thank you.

youikim commented 3 years ago

solved

jimmy10023 commented 3 years ago

Hi, I'm having the same problem, how did you solve it?