Improve scraper for Institute of Education Sciences (Phase 2) - Githubissues

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

Improve scraper for Institute of Education Sciences (Phase 2) #167

Closed higorspinto closed 4 years ago

higorspinto commented 4 years ago

During phase 1 we created a functional scraper for crawling and parsing data from this office. The scraped data was successfully ingested into the data portal.

For phase 2, we need to improve the quality of metadata and data-content for the datasets being generated by the scraper.

https://ies.ed.gov/

Acceptance Criteria

[x] we have marked improvement in the quality of metadata and data-content of datasets produced by the scraper.
[x] the improved quality datasets are visible on the data portal

Tasks

[x] Ensure datasets produced have a description metadata
[x] Ensure datasets have a publisher metadata
[x] Improve other metadata (use defaults where available)

higorspinto commented 4 years ago

Related PRs:

https://github.com/CivicActions/edscrapers/pull/201 https://github.com/CivicActions/edscrapers/pull/173