(New Office) Office of Legislation and Congressional Affairs: create scraper for this office (Phase 1) - Githubissues

CivicActions / edscrapers

US Department of Education Data Scraping Kit; see https://us-ed-scraping.ckan.io/dataset

GNU Affero General Public License v3.0

15 stars 9 forks source link

(New Office) Office of Legislation and Congressional Affairs: create scraper for this office (Phase 1) #160

Closed higorspinto closed 4 years ago

higorspinto commented 4 years ago

The Office of Legislation and Congressional Affairs is among the list of new offices whose datasets need to be ingested into the data portal. For this to happen, we need to create a new scraper to crawl/parse the available webpages of the office.

https://www2.ed.gov/about/offices/list/olca/index.html

Acceptance Criteria

[ ] We have a functional crawler that crawls through the webpages of the offices
[ ] We have a functional parser that understands the page structures and generates structured data
[ ] Datasets are produced when our scraper is run

Tasks

[ ] Identify the possible page structures in the target site
[ ] Write one or multiple parsers that cover as many cases as possible
[ ] Test if it runs well within the pipeline