Add cloud function code for uploading files from url to gcs

SatcherInstitute / prototype

MIT License

4 stars 4 forks source link

Add cloud function code for uploading files from url to gcs #15

Closed jenniebrown closed 3 years ago

jenniebrown commented 3 years ago

This is a rough copy of Brian's demo cloud function code with error handling that can be generically used for pulling data from a url #14

jenniebrown commented 3 years ago

Actually, maybe we should put the python file under a directory named something like "ingestion" so by functionality instead of by compute mechanism.

done

jenniebrown commented 3 years ago

Not blocking this change, but is there a way to get a last updated timestamp for the data? That would be useful to check before copying it over every time if it hasn't changed.

Yeah, we will definitely want that. The USDA site has an html table with the last updated date but I'm looking to see if it's possible to do that in a generic way.