search

ScrapeGraphAI / Scrapegraph-ai

Python scraper based on AI
https://scrapegraphai.com
MIT License
14.35k stars 1.17k forks source link

Scraping dynamic content #279

Open JackyZhang1219 opened 3 months ago

JackyZhang1219 commented 3 months ago

Is your feature request related to a problem? Please describe.

Is it possible to get all the dynamically loaded content from a url? E.g. getting all the reviews from IMDB requires clicking "load more" over and over again, and Scrapegraph-ai currently only gets the reviews loaded first

VinciGit00 commented 3 months ago

not yet

PeriniM commented 3 months ago

Hey @JackyZhang1219 thanks we'll look into It! if you have some suggestions write them in the Discord channel

hogking commented 3 months ago

This is a great feature request. or example, get a 'next page' button to jump to the next list page, like scrapy-slash?

angelotc commented 1 month ago

This is a scraper, not a crawler. The way I see it is you can integrate this with scrapy.