oese-parser: oese-parser P7 complete

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

oese-parser: oese-parser P7 complete #48

Closed osahon-okungbowa closed 4 years ago

osahon-okungbowa commented 4 years ago

Created one parser for OESE
Adjusted scrapy config to turn off http caching (this enabled the retrieval of OESE pages)
Adjusted crawler by adding allowed_domains

osahon-okungbowa commented 4 years ago

Changes made as requested. The process_value function came when i was investigating why the crawler was not being triggered initially. Commented out the function call when i discovered the cache was responsible. However, forgot to delete the function. All done now. Thanks