search

calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros
https://calogica.github.io/dbt-expectations/
Apache License 2.0
1.04k stars 126 forks source link

Support ephemeral models in expectations #176

Closed judahrand closed 2 years ago

judahrand commented 2 years ago

Current ephemeral models do not work with some tests provided by dbt-expectations. This PR attempts to support them through the use of adapter.adapter.get_columns_in_query.

clausherther commented 2 years ago

Hey @judahrand thanks for this PR! I'll take a more detail look, but for now looks like the integration tests are failing with error 'obj' is undefined (at least on postgres), using dbt 1.1.0.

Not sure if you can see this, but for reference: https://app.circleci.com/pipelines/github/calogica/dbt-expectations/57/workflows/61b9a7e6-d7be-4402-90ad-ea7de70006c1/jobs/49?invite=true#step-104-550

judahrand commented 2 years ago

I'll have a look when I get a chance - I think it's a copy pasta error

judahrand commented 2 years ago

@clausherther I think I fixed the error are you able to allow the integration tests to run again?

clausherther commented 2 years ago

@judahrand looks like we still got some database errors as well as test failures https://app.circleci.com/pipelines/github/calogica/dbt-expectations/58/workflows/0d2372ac-5bd1-443a-9d9e-95c8b1ca5c36/jobs/50?invite=true#step-104-370

clausherther commented 2 years ago

@judahrand I think the issue is that the ephemeral model still returns {'type': 'PostgresRelation'}, {'type': 'SnowflakeRelation'} etc, just like the table materialization. I think you'll want to use model.is_cte in this case, see https://docs.getdbt.com/reference/dbt-classes#using-relations However, I get this error when dbt is trying to get columns for a CTE:

 'dbt.context.providers.RuntimeDatabaseWrapper object' has no attribute 'get_columns_in_query'

Are you sure this is supposed to work for ephemeral models/CTEs?

clausherther commented 2 years ago

I don't think you can do column introspection on ephemeral models. See: https://github.com/dbt-labs/dbt-utils/blob/main/macros/cross_db_utils/_is_ephemeral.sql

clausherther commented 2 years ago

Hey @judahrand ! Just wanted to check in if this is something you're still pursuing or if we should close this PR? Thanks!