Automated scraping markup+CSS from a list of relevant URLs, using a variety of user-agent strings. Provides reporting on usage of CSS properties and apparent user-agent sniffing.
Update scrapy version to 0.17 so we can use a better httpcache middleware.
Update every file where scrapy.conf.settings was used as a singleton since the api has changed
Update scrapy version to 0.17 so we can use a better httpcache middleware. Update every file where scrapy.conf.settings was used as a singleton since the api has changed