Automated scraping markup+CSS from a list of relevant URLs, using a variety of user-agent strings. Provides reporting on usage of CSS properties and apparent user-agent sniffing.
I noticed that Scrapy's dependencies weren't listed in compiled.txt or pure.txt or installed in the vendor lib; this adds them.
I also moved pytest and other dev-only deps out to dev.txt; it's not useful to have that stuff in the vendor lib since scripts (like the "py.test" script) aren't installed for stuff in the vendor lib.
I noticed that Scrapy's dependencies weren't listed in
compiled.txtorpure.txtor installed in the vendor lib; this adds them.I also moved pytest and other dev-only deps out to
dev.txt; it's not useful to have that stuff in the vendor lib since scripts (like the "py.test" script) aren't installed for stuff in the vendor lib.