Ope collections sources

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

Closed osahon-okungbowa closed 4 years ago

osahon-okungbowa commented 4 years ago

TASKS TACKLED

fixed metadata request headers retrieval
resource headers now available in datajson transformer output
fixed backtracking issue caused by 'false' referer head AKA bug where collections and sources are the same
implemented simple throttling for requests of resources which are outside of scrapy's control. this solves rate limit errors
other performance and resilience improvements

osahon-okungbowa commented 4 years ago

@nightsh 2 collections, 1 source in ocr now. Pls kindly merge by tomorrow morning. so P4 (OELA)can benefit from all improvements. Thank you :+1: