search

biglocalnews / warn-scraper

Command-line interface for downloading WARN Act notices of qualified plant closings and mass layoffs from state government websites
https://warn-scraper.readthedocs.io
Apache License 2.0
28 stars 10 forks source link

Small DC scraper problems #633

Open stucka opened 3 months ago

stucka commented 3 months ago

The scraper assumes a link for 2014 is listed but echoes a different year. For 2024's index, at least, that is not the case; no link to 2014 is offered, and therefore the scraper will not automatically try to download 2014.

Scraper grabs old HTML even when there's no update, e.g., in 2024 maybe we want to update 2024 and 2023, but we don't need to grab 2019 again. A small optimization might be to try to hit older URLs with utils.fetch_if_not_cached and then read from it the files with cache.

stucka commented 3 months ago

I'm sure there was a reason to use this uuid thing instead of labeling the files with the years, but I'm not seeing it at the moment. =)

stucka commented 3 months ago

Also should maybe nix the http:// prefix in the use of 2014.