codeforboston / maple

MAPLE makes it easy for anyone to view and submit testimony to the Massachusetts Legislature about the bills that will shape our future.
https://mapletestimony.org
MIT License
38 stars 106 forks source link

Modify eventScraper to pull in DocumentHistoryActions instead of Hearings #1509

Open mvictor55 opened 2 months ago

mvictor55 commented 2 months ago

Modify scrapeEvents function to pull in DocumentHistoryActions instead of Hearings.

Currently, the scrapeEvents function is making use of the listEvents function from the malegislature file.

See:

https://github.com/codeforboston/maple/blob/main/functions/src/malegislature.ts https://github.com/codeforboston/maple/blob/main/functions/src/events/scrapeEvents.ts

iWumboUWumbo2 commented 2 months ago

The backbone for publishing notifications currently already exists in the publishNotifications cloud function. However, no notifications are published because the events it looks for are in the form bill-example-1 or org-example-1.

Right now, the events scraper (scrapeEvents) only scrapes hearings (scrapeHearings) and sessions (scrapeSessions) which are saved in the form hearing-1 and session-1-1, respectively. This needs to be expanded to scrape all DocumentHistoryActions and store them in a form that can be understood by publishNotifications.

  1. Create a new scraper class (DocumentHistoryActionsScraper) that is a child of the EventsScraper abstract class that all of the other scrapers inherit from (see: scrapeEvents.ts). Use the HearingScraper as reference.
  2. Add the scraper to the existing list of scrapers (see: runScrapers.ts)
sashamaryl commented 2 months ago

the documenthistoryactions are called the "bill history" and it is scraped and saved as part of the bill info, used on the bill page, right now in the bill history modal. hopefully this is helpful--

https://github.com/codeforboston/maple/blob/main/functions/src/bills/bills.ts

image