Run pudl ETL on dask kubernetes

catalyst-cooperative / pudl

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

https://catalyst.coop/pudl

MIT License

481 stars 110 forks source link

Run pudl ETL on dask kubernetes #905

Open rousik opened 3 years ago

rousik commented 3 years ago

This is one of the supported prefect run environments and would probably make the most sense for us. Figure out what needs to be done in order to get this working.

rousik commented 3 years ago

Preemptible instances are way cheaper, this document outlines how to ask for preemptible instances when creating k8s nodes:

https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms