search

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.
https://catalyst.coop/pudl
MIT License
472 stars 108 forks source link

Create terraform project for nightly builds #1785

Open bendnorman opened 2 years ago

bendnorman commented 2 years ago

I set up Nightly Builds using a combination of gcloud CLI and UI. The downside of this is that the setup isn't well documented. Terraform/infrastructure as code solves this issue by declaratively stating desired cloud resource configuration.

This might not be necessary if nightly builds change a lot as we begin to use dagster cloud. We will be using cloud resources to a certain degree and should use terraform!

zaneselvans commented 1 year ago

@bendnorman @jdangerx Did this happen? I saw some terraform stuff floating around.

bendnorman commented 1 year ago

Not yet. @jdangerx used terraform to set up WIF #2259. I could see us revisiting nightly builds after the dagster conversion to use terraform and branch deployments.

bendnorman commented 1 month ago

@jdangerx is it worth bring out Batch setup into terraform or leave it as is for now?

jdangerx commented 1 month ago

Good question! Our current batch setup is sort of under IaC already, as in it's generated in generate_batch_config.py. I'm not sure if we get that much from Terraforming it since it seems like Terraform would also be jamming a bunch of stuff into a JSON that we submit to the Batch API. Let's leave it as is for now.