cagov / data-orchestration

Orchestration tooling for the CalData Data Services and Engineering team
MIT License
0 stars 0 forks source link

Normalize project IDs #15

Closed ian-r-rose closed 1 year ago

ian-r-rose commented 1 year ago

I've been a bit sloppy in hard-coding project IDs into some of the scheduled jobs here (e.g., when writing into a particular BigQuery dataset). That makes this project hard to lift from one project to another (including when I'm doing some testing in a sandbox account).

Probably better to store default project IDs in an airflow variable and pick it up where appropriate. Variables could be stored in a plain text file and re-sync'd on deploy.