(New Office) Institute of Education Sciences: create scraper for this office (Phase 1)

[higorspinto]

Description

Institute of Education Sciences (IES) is among the list of new offices whose datasets need to be ingested into the data portal. During our prior scraping run we had the NCES scraper, NCES is a suboffice/sub-organisation of IES. Therefore, we need to update/improve the current NCES scraper to cater for IES and all its suboffices (including NCES).

https://ies.ed.gov/

Acceptance Criteria

• [x] We have a functional crawler that crawls through the webpages of the offices
• [x] We have a functional parser that understands the page structures and generates structured data
• [x] Datasets are produced when our scraper is run

Tasks

• [x] Identify the possible page structures in the target site
• [x] Write one or multiple parsers that cover as many cases as possible
• [x] Test if it runs well within the pipeline

Jira Card

[georgiana-b]

Solved in https://github.com/CivicActions/edscrapers/pull/173.

We ended up adapting the existing NCES parser to also handle the results on https://ies.ed.gov/pubsearch/index.asp since the page structure was very similar to that of https://nces.ed.gov/pubsearch/index.asp.