search

datamade / court-scrapers

MIT License
2 stars 0 forks source link

Add court call scrape #38

Closed antidipyramid closed 6 months ago

antidipyramid commented 6 months ago

Overview

This PR adds court call scraping to the nightly scrape action.

Notes

Preventing duplicate court call rows

Court calls are updated nightly so we'll be scraping a single court call multiple times over the course of a few days. I added a hash column to the court_call table to ensure that we don't have duplicates.

Not-yet scraped court cases

Court calls scrapes include cases that might not be in the database yet. The court_call table still have a foreign key to the court_case table but not all of the rows in court_call will map to a valid court case.

antidipyramid commented 6 months ago

@fgregg I also pulled out the court call scrape into its own action since it can take hours.