search

hydrosquall / yale_athletics_data

data about student-athletes at harvard and yale from 1940 - 2016
MIT License
4 stars 2 forks source link

Scrapy Spider #6

Open hydrosquall opened 8 years ago

hydrosquall commented 8 years ago

Consider making a scrapy spider rather than using asyncio / requests/ bs4 combo to visit the 24k+ athlete bios. It can then be deployed to scrapinghub as long as it will complete within 24 hrs.