search

biglocalnews / warn-scraper

Command-line interface for downloading WARN Act notices of qualified plant closings and mass layoffs from state government websites
https://warn-scraper.readthedocs.io
Apache License 2.0
29 stars 10 forks source link

CA scraper hopelessly inefficient #652

Open stucka opened 3 months ago

stucka commented 3 months ago

The scraper is ... strange. Right now every time it downloads an Excel file of recent filings. Every time it also re-downloads a bunch of historic PDFs,one of which isn't parsing correctly now but has data already in the Excel file.

We can't possibly know if the scraper will fail on when the next generation of PDF is released.

New workaround to at least bypass some of that: ZIP up BLN's existing cache archive (through ca branch) and include the output CSV in the ZIP.

Throw that up in Google Cloud Storage.

Take the last good CSV and put that separately in Google Cloud Storage.

Rework the scraper to not download any of those existing PDFs through today's dates.

Rework the scraper to download and import the processed data from Google Cloud Storage. Import it into the scraper before writing the final output CSV.

This may break as soon as the scraper detects the first July 2024 layoffs and tries to parse that PDF, but at least some of the runtime, data transfer and senseless processing can be avoided.

stucka commented 2 months ago

The scraper's PDF parsing code contained a fatal flaw that would have blocked any future success, and I believe I have that patched up with https://github.com/biglocalnews/warn-scraper/pull/653

The inefficiencies., however, remain. We're downloading PDFs that haven't changed in a decade.