Open hagai-arad opened 11 months ago
Hey @hagai-arad ! Thanks for raising this. We'll investigate what may be happening here.
This is working as intended.
We don't support config substitutions for arbitrary config fields for the Fluent Datasources. We only do it for things like passwords/credentials, connection_string
etc.
I don't think this is well-documented.
This might be something we could add support for as part of a new feature request.
@Kilo59 Thanks. Should I open a new feature request? Not sure how it works
@hagai-arad yes, sorry for the slow response. Or you could edit this issue and title to highlight that it's a feature request.
It should be simple to implement. If you're interested, I can guide you to implement this yourself.
@Kilo59 Hey, I would like to get your guidance. How can we proceed from here? Thanks!
Sorry again @hagai-arad I missed the notification that you responded.
Our Datasource and DataAsset classes are pydantic models.
You'll need to update the annotation on the TableAsset.name
field to be a Union[ConfigStr, str]
this will cause pydantic to coerce the value as a ConfigStr
and only if it cannot (there's no ${}
syntax) read it as a normal string.
https://docs.pydantic.dev/1.10/usage/types/#unions
That should be enough by itself to get your desired behavior.
Hi @Kilo59, Is this issue still up for grabs? If so I'd be happy to give it a go.
I have a similar use case where I want to replace the project_id
of a query with a value in the config.yaml
:
bigquery_datasource:
type: sql
assets:
customer:
type: query
order_by: []
batch_metadata: {}
query: select * from ${project_id}.dataset.customer
From what I can tell I would have to make the same annotation change you suggest above here? https://github.com/great-expectations/great_expectations/blob/25dd66ea30d865f0c994ac50e183bef60d70c3cc/great_expectations/datasource/fluent/sql_datasource.py#L805
Describe the bug When using a substitution syntax in
great_expectations.yml
(e.g.${my_value}
) in the table_name attribute of a data asset - it doesn't substitute by the env var value nor the value given inuncommitted/config_variables.yml
. Instead, it uses the string itself as the table name when running queries against this data asset:select count(*) from ${table_name}
.To Reproduce great_expectations.yml config:
The executed code:
Full stack trace:
Expected behavior The
${TABLE_PREFIX}
should have been replaced with the value stored inuncommitted/config_variables.yml
or by the value of theTABLE_PREFIX
env var. The substitution works in all other places in my great_expectations.yml file (e.g. connection_string).Environment (please complete the following information):