How to deal with pages with infinite scroll ?

nrabinowitz / pjscrape

A web-scraping framework written in Javascript, using PhantomJS and jQuery

http://nrabinowitz.github.io/pjscrape/

MIT License

996 stars 159 forks source link

Open geekonweb opened 11 years ago

geekonweb commented 11 years ago

Not sure how to handle pages with infinite scroll feature where a window scroll event is used to trigger a next page load.

One example could be below where next page gets loaded after you scroll down to bottom: http://www.shopbychoice.com/mobile-phones/smartphones

pavankat commented 9 years ago

Maybe you can scroll to the bottom using jQuery and then do a set time out and wait 15 seconds, then repeat 100 times, and then scrape