datamade / court-scrapers

MIT License
2 stars 0 forks source link

Add runtime check to nightly scrape #61

Closed antidipyramid closed 4 months ago

antidipyramid commented 4 months ago

It looks like the nightly case scrape is still running up against the 6 hour github actions time limit (perhaps because of case backfilling?).

This PR adds logic to keep track of scrape runtime. The nightly scrape should stop if we've been scraping for 5.5 hours (we'll leave 30 minutes for cleanup and deployment).

Part of #59