(New Office) Office of Finance and Operations: create scraper for this office (Phase 1)

[higorspinto]

The Office of Finance and Operations is among the list of new offices whose datasets need to be ingested into the data portal. For this to happen, we need to create a new scraper to crawl/parse the available webpages of the office.

https://www2.ed.gov/about/offices/list/ofo/index.html

Acceptance Criteria

• [ ] We have a functional crawler that crawls through the webpages of the offices
• [ ] We have a functional parser that understands the page structures and generates structured data
• [ ] Datasets are produced when the scraper is run

Tasks

• [ ] Identify the possible page structures in the target site
• [ ] Write one or multiple parsers that cover as many cases as possible
• [ ] Test if it runs well within the pipeline
• [ ] Ensure datasets produced have a description metadata
• [ ] Ensure datasets have a publisher metadata
• [ ] Improve other metadata (use defaults where available)

Jira Card