Add default protection against billing surprises

brabster / dbt_bigquery_template

A repository template to use with dbt_core

MIT License

1 stars 0 forks source link

Open brabster opened 9 months ago

brabster commented 9 months ago

As a consumer I want interactions with BigQuery to fail if they scan excessive data So that I am protected from unexpected costs

ACs:

default query scan limit set to 10GB (can do 100 such queries before exceeding the 1TB free tier)
scan limit is independently configurable in local setup and in the build so that I can relax the guardrail when I need to

brabster commented 8 months ago

PITA to calculate bytes for a sensible unit - let Jinja do the work

maximum_bytes_billed: "{{ env_var('DBT_MAX_GIGABYTES_BILLED', '50') | int * (1024 * 1024 * 1024) }}" # 50GB billable bytes estimate limit

DBT_MAX_GIGABYTES_BILLED: ${{ vars.DBT_MAX_GIGABYTES_BILLED }} in build