search

dviator / WeSaidSheSaid

For the development of an application to store, document, and analyze political speeches and the media's interpretation of them.
1 stars 0 forks source link

Crawler needs to get to 'show more videos' section of search #8

Closed dviator closed 9 years ago

dviator commented 9 years ago

The crawler should follow this link repeatedly and crawl the results from those pages as well. Following this down to the end should provide coverage of most of what's needed.

dviator commented 9 years ago

Example of using selenium to click a button and make javascript calls, integrated with scrapy. http://stackoverflow.com/questions/17975471/selenium-with-scrapy-for-dynamic-page

dviator commented 9 years ago

Another example: http://stackoverflow.com/questions/10648644/executing-javascript-submit-form-functions-using-scrapy-in-python

dviator commented 9 years ago

Now completed via Selenium