search

SatcherInstitute / prototype

MIT License
4 stars 4 forks source link

Data Ingestion - Household income #14

Closed mayaspivak closed 3 years ago

mayaspivak commented 3 years ago

Create a scheduler job to get household income data into GCS bucket

https://data.ers.usda.gov/reports.aspx?ID=17828

jenniebrown commented 3 years ago

Added a cloud scheduler job to run every Thursday in the test project called household-income-scheduler. According to the USDA website, this data is only updated once per year. Running once per week will help catch issues if the data source is moved, for example.

slagathorr commented 3 years ago

Can you point me to where this schedule is? I'm starting to put together the Terraform scripts and it would make sense to include this.

jenniebrown commented 3 years ago

Sure, it's in the fellowship-test-internal project named household-income-scheduler. I added it through the UI so I don't have any configuration elsewhere.

mayaspivak commented 3 years ago

This issue will now also include a function/service to get the data from GCS into bigquery and any data standardization that needs to be done within bigquery.