dbt-labs / spark-utils

Utility functions for dbt projects running on Spark
https://hub.getdbt.com/fishtown-analytics/spark_utils/latest/
Apache License 2.0
31 stars 15 forks source link

Add support for get_relations_by* / get_tables_by* using information_schema in Unity Catalog #28

Open tfayyaz opened 2 years ago

tfayyaz commented 2 years ago

Databricks and Unity Catalog now supports information_schema https://docs.databricks.com/sql/language-manual/sql-ref-information-schema.html and so it should now be possible to support some of the missing shims listed in the readme and as one of the tasks in https://github.com/dbt-labs/spark-utils/issues/7:

The missing shims are:

I will be willing to try and contribute these but will need guidance on how to define the logic that only allows these to be compatible if you are using >=dbt-databricks==1.1.1 and have Unity Catalog enabled.

Thanks Tahir