Integrate Page Switching Method into Main Scraper Method

WayneRose-95 / Metacritic_Webscraper-

A side project on creating my own first web scraper.

GNU General Public License v3.0

0 stars 0 forks source link

Closed WayneRose-95 closed 2 years ago

WayneRose-95 commented 2 years ago

The page switching method.

Find a way to implement it into the main scraper method.

Maybe use a nested for loop?

WayneRose-95 commented 2 years ago

Sample scraper now clicks next page and continues scraping the website. However, it does stop after a certain point.

Maybe, an extra time.sleep() needs to be added to allow the scraper to find the next pages without throttling the server too much.

Sample Scraper New Additions