search

jamesturk / scrapeghost

👻 Experimental library for scraping websites using OpenAI's GPT API.
https://jamesturk.github.io/scrapeghost/
Other
1.42k stars 86 forks source link

pagination #52

Closed daonsh closed 1 year ago

daonsh commented 1 year ago

If the website I scrape needs pagination to get all the data, can this tool click on "next" (or similar) and continue with the next page?

jamesturk commented 1 year ago

see PaginatedSchemaScraper. https://jamesturk.github.io/scrapeghost/usage/

you may want to extend this yourself depending on how the pagination works but it's a starting place.