P3 Crawling: Office of Postsecondary Education - Githubissues

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

P3 Crawling: Office of Postsecondary Education #5

Closed Daniellappv closed 4 years ago

Daniellappv commented 4 years ago

Description: Scrape metadata for https://www2.ed.gov/about/offices/list/ope/index.html

Acceptance criteria

[x] We have a data dump with all the resources metadata we can get from target site

Task-list:

[x] Crawl the site
[x] Perfect the crawling to reach as many resources as possible
[x] Integrate with the existing pipeline rules (provide a HTML response for the parser)
[x] Test run with a dummy parser - it should collect datasets and dump them into JSON files
[x] Push the code once it checks all the above criteria

Jira card: https://open-data-ed.atlassian.net/browse/OD-501