Added a DocumentCloud uploader - Githubissues

data-liberation-project / aphis-inspection-reports

Inspection data and PDFs from the USDA's Animal and Plant Health Inspection Service.

13 stars 3 forks source link

Added a DocumentCloud uploader #11

Closed palewire closed 1 year ago

palewire commented 1 year ago

For #9

Documents go here: https://www.documentcloud.org/app?q=%2Bproject%3Ausda-aphis-inspection-rep-211004%20

I added your gmail as an admin.

Three environment variables would need to be added to the repository. You can see them in the code.

jsvine commented 1 year ago

Thanks! A couple of questions:

Per #9, I'd like to use the actual inspection IDs rather than the hash-derived filenames. I can do that part soon.
What do you think of the proposal in #9 to track uploaded PDFs locally, rather than querying DocumentCloud for the list of filenames each time? I'm thinking that, with 80k+ reports, it might go somewhat slowly ... though that's just a guess, not having tested it. Or not actually much of a problem?

palewire commented 1 year ago

Local cache is fine with me. You're probably right about 80k being too many to query routinely.

jsvine commented 1 year ago

Sounds good. Happy to take it from here, based on the solid DocCloud groundwork you've laid.

jsvine commented 1 year ago

Pushed my proposed tweaks, with a few examples uploaded for demonstration.

The failing mypy check here is stumping me. Same command works fine locally, and I don't see the actual error cited in the logs.

jsvine commented 1 year ago

Ahh, thank you — I see what the issue is now and have push a (hopeful) fix.

palewire commented 1 year ago

I made a couple tiny, cosmetic tweaks. Looks good to me. Let's upload these fools.

jsvine commented 1 year ago

Rad, merging. Will track upload progress in https://github.com/data-liberation-project/aphis-inspection-reports/issues/9.