konklone
commented
10 years ago This really only works if the entire IG is being archived at once. For example, if ./inspectors/usps.py were being run with the default arguments, it would only cover the current year, 2014, and so the built-up dict would only verify that the current year's report_id was unique.
I think the right approach is as @parkr suggests, that it analyze the state of the filesystem. I had envisioned this as a separate task, that runs holistically against the entire current contents on disk, and not something run during the scrape. In some ways, it's easier architecturally to just build it in to the scrape, and make it known that by running an archival scrape you can get this behavior -- but it also breaks the implicit guarantee of idempotency that the scrapers otherwise have. In other words, you could run a scraper once with some group of args (X), and it's fine, then run again with args Y, and it's fine, but then run it with args X, and suddenly there's an error now.
I think there's room in the architecture for a separate validation/analytics script, that looks at the contents of disk and gives you some advice. Analyzing uniqueness of report_ids is a great place to start with that.
divergentdave
Looks great! Might be worth fetching them from the filesystem if possible, too. :+1: