search

alirezamika / autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python
MIT License
6.16k stars 648 forks source link

Pagination #47

Closed programmeddeath1 closed 3 years ago

programmeddeath1 commented 3 years ago

Hi how can I handle pagination for example if I want to fetch comments and reviews. And is there a way to detect/handle consecutive pages other than by listing them like how general scrapers would have a click function to move to different pages or actions.

go-delicious commented 3 years ago

There are two ways you can do it. If the site has a next button, you can scrape the button and see if it is active, if it's active you add one to the page number in the URL and carry on scrapping.

Two is you do an Initial scrape to get the total number of pages. Then do a range loop using the number of pages to scrape each page.