search

alirezamika / autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python
MIT License
6.16k stars 648 forks source link

Is there a way to scrape information from multiple pages? #59

Closed siddharthchauhan closed 3 years ago

siddharthchauhan commented 3 years ago

I want to scrape information from all the pages of a website, Is there a way that it will go to next page automatically and scrape it?

go-delicious commented 3 years ago

Are you trying to scrape all the site or just scrape the next page until there is no next page?

siddharthchauhan commented 3 years ago

Next page until there is no page.

go-delicious commented 3 years ago

Writing a recursive function should handle that.

When you scrape, also get the next button, if it's active you call the same function using the link of the button.

siddharthchauhan commented 3 years ago

Can you please write the psudo code or something how will I fetch URL for the next page.

siddharthchauhan commented 3 years ago

The page i want to scrape has no next button it has symbol '>'.

siddharthchauhan commented 3 years ago

The website has this kind of structure < 1, 2, 3 ,4, 5 >

go-delicious commented 3 years ago

The one you are currently on should look different to the rest.