search

dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
400 stars 228 forks source link

[ADAP-522] Three-part identifiers (`catalog.schema.table`) #755

Open dbeatty10 opened 1 year ago

dbeatty10 commented 1 year ago

Spark >= 3.0 can discover tables/views from multiple catalogs, such as a Hive or Glue catalog. This is a prerequisite to enable so-called three-part identifiers for dbt-spark (catalog.schema.table).

To keep PRs concise for any refactoring plus the implementation, we are splitting this over multiple issues.

### Three-part identifiers
- [ ] https://github.com/dbt-labs/dbt-spark/issues/751
github-actions[bot] commented 1 year ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 12 months ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

mukeshkumarkulmi commented 6 months ago

Hi Everyone, Can someone suggest how to use Apache Nessie catalog with Spark adapter in dbt. If we modify generate_schema_name to add nessie catalog name, then listNone* API fails as it creates *show table extended in nessie. like ''* query to get table names in the schema. And we cannot use "generate_database_name" for customisation in Spark adapter as it fails with "Cannot set database in spark!" error. Also, in case of incremental models, DBT only creates insert into command, if the table already exists in the schema when it run "show table extended" query but show table extended in nessie. like '' this query is failing, so DBT always create new table in default catalog. Please suggest.