P6 Crawling: Office of Planning, Evaluation and Policy Development - Githubissues

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

P6 Crawling: Office of Planning, Evaluation and Policy Development #8

Closed Daniellappv closed 4 years ago

Daniellappv commented 4 years ago

Description: Scrape metadata for https://www2.ed.gov/about/offices/list/opepd/index.html?src=oc

Acceptance criteria

[x] We have a data dump with all the resources metadata we can get from target site

Task-list:

[x] Crawl the site
[x] Perfect the crawling to reach as many resources as possible
[x] Integrate with the existing pipeline rules (provide a HTML response for the parser)
[x] Test run with a dummy parser - it should collect datasets and dump them into JSON files
[x] Push the code once it checks all the above criteria

Jira card: https://open-data-ed.atlassian.net/browse/OD-506