search

calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros
https://calogica.github.io/dbt-expectations/
Apache License 2.0
1.01k stars 123 forks source link

[Feature Request] support Databricks #244

Closed frankivo closed 1 year ago

frankivo commented 1 year ago

Is your feature request related to a problem? Please describe. Most test, but not all, work with Databricks. For example: expect_column_values_to_match_regex

Describe the solution you'd like A valid implementation for those missing tests.

Describe alternatives you've considered I'm writing custom tests for now.

Additional context Willing to do some coding myself :)

clausherther commented 1 year ago

Hi @frankivo - we don't have a way to test against databricks - or any platform other than Postgres, BigQuery or Snowflake - in our CI pipeline, so we don't currently support platforms outside of that core set. However, as I've outlined in the answer to this question you can add support for other platforms via shims in their respective **-utils package, such as spark-utils. Hope that helps!

frankivo commented 1 year ago

I'll try :) https://github.com/databricks/dbt-databricks/pull/280

clausherther commented 1 year ago

@frankivo I wonder if they'll want you to implement this in spark-utils though, not in the adapater repo?

frankivo commented 1 year ago

Afaik I'm using a databricks feature and not a spark feature. We'll see.

clausherther commented 1 year ago

I don't think there is a databricks-utils package though, so spark-utils seems to be where the databricks/sparks macros go. My guess would be the macros in the adapter mirror what dbt-core supports, and since they don't have a regexp implementation, the databricks adapter shouldn't either.