search

biglocalnews / warn-scraper

Command-line interface for downloading WARN Act notices of qualified plant closings and mass layoffs from state government websites
https://warn-scraper.readthedocs.io
Apache License 2.0
30 stars 10 forks source link

Build automated QA checks #598

Open stucka opened 10 months ago

stucka commented 10 months ago

As @Kirkman found in #597 , a scraper can stop producing output without triggering an error in workflow.

While a few states keep WARN and non-WARN layoffs in the same database it's unlikely that many states would ever have a reduction in the number of incidents reported in the files getting scraped. So if a state moves from 283 reports to 123 reports or 0 reports that should get flagged. Simple row counts of CSVs compared to earlier snapshots would have caught Missouri problem.

Weekly Github Action built in warn-support repo, perhaps?

chriszs commented 8 months ago

Looks like how to approach this problem is being considered over in https://github.com/biglocalnews/warn-transformer/issues/236