GoogleCloudPlatform / cloud-data-quality

Data Quality Engine for BigQuery
Apache License 2.0
258 stars 57 forks source link

Allow environment variables substitition in yaml files #220

Open Luiscri opened 1 year ago

Luiscri commented 1 year ago

It would be useful if environment variables substitution was allowed for yaml files. An usage example would be refering to a different table according to the environment we are:

entities:
  MY_TABLE:
    source_database: BIGQUERY
    table_name: ${GOOGLE_CLOUD_PROJECT}.mydataset.mytable
...

# or

rule_bindings:
  MY_BINDING:
    entity_uri: bigquery://projects/${GOOGLE_CLOUD_PROJECT}/datasets/mydataset/tables/mytable
...
thinhha commented 1 year ago

Hi @Luiscri thanks for the suggestion. I think this would be a great feature. I estimate this effort to be quite large as we will need to make a few important decisions on how to allow interpolating environment variables into the YAML configs. I will update this ticket when we are able to prioritize this work.

victorbrub commented 5 months ago

That would be really helpful for my project actually. There's a plan for this implementation?