dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://cloud.google.com/dataform/docs
Apache License 2.0
858 stars 166 forks source link

Be able to pass parameters to dataform run command #1175

Closed Altons closed 3 years ago

Altons commented 3 years ago

As I data engineer I would like to be able to pass parameters to a pipeline in bigquery as

dataform run --tags countryReport --vars {country:"UK", department:"marketing"}

which will inject the parameters in my sqlx as

select * from 
 ${ref("report_mart")}
 where country = ${vars.country} and department =  ${vars.department}

and compiled version would look like this:

select * from 
 `my_project.my_dataset.report_mart`
 where country = "UK" and department =  "marketing"

This will enable the creation/update of tables reusing the same logic.

BenBirt commented 3 years ago

I don't love the JSON blob model because bash makes it a bit of a pain to use. I'd suggest --var foo=bar --var baz=qux --vars=foos=bars,bazs=quxs