binux / pyspider

A Powerful Spider(Web Crawler) System in Python.
http://docs.pyspider.org/
Apache License 2.0
16.48k stars 3.69k forks source link

Automatic loading is invalid when use patomjs parse webpage #192

Open wsshopping opened 9 years ago

wsshopping commented 9 years ago

https://github.com/binux/pyspider/issues/new#fullscreen

when i use the patomjs parse the webpage http://www.zhanqi.tv/games/lol, load next page is invalid, more detalis as follows: you can use this enviroment : http://demo.pyspider.org/debug/dengxiaobing

i want to scrap the total list of lol, but the result is just 30 ,and i run the scripts in web console, it works fine . that is : setInterval("window.scrollTo(0,document.body.scrollHeight)",5000);

binux commented 9 years ago

Try to change the height of page view in pyspider/fetcher/pantomjs_fetcher.js

page.viewportSize = { width: 1024, height: 768000 }

to a smaller size, 768 for instance.