[Feature] Add support for unity catalog in source config

[data-blade]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

setting sources in dbt-databricks works like this:

sources:
  - name: <name>
    catalog|database: <catalog>
    schema: <schema>

dbt-spark should support unity in the same way

Describe alternatives you've considered

in dbt-spark, using...

database: <catalog>
schema: <schema>

will run into this error.

and setting...

catalog: <catalog>
schema: <schema>

will fail too, since catalog is unexpected.

the only working setting is

schema: <catalog>.<schema>

maybe only for us: the above prevents the parallel use of dbt-databricks adapter (which we require), so we overcome this by

database: "{{ none if target.type == 'spark' else '<catalog>' }}"
schema: "{{ '<catalog>.<schema>' if target.type == 'spark' else '<schema>' }}"

Who will this benefit?

all clients using dbt-spark adapter on databricks' unity catalog (to not use hacks anymore)

Are you interested in contributing this feature?

if time allows

Anything else?

No response

[amychen1776]

Hi @data-blade ! We recommend that users on Databricks use dbt-databricks rather than dbt-spark. Would you be able to clarify why you need to use dbt-spark?

[data-blade]

to run production on job clusters to save cost.

if dbt-databricks could connect to job clusters, i'd be ecstatic for a week!

[amychen1776]

Ah I see - they have a PR open to operate on serverless job clusters that are getting released soon so I think you should take a look over there :)

That said - I am going to close this ticket for now because this is not something we are going to tackle in our near term roadmap, especially as our recommendation is for Databricks customers to use the dbt-databricks adapter.