[Feature] Databricks Compatibility

[fivetran-catfritz]

Copied from fivetran/dbt_mixpanel #34.

Is there an existing feature request for this?

• [X] I have searched the existing issues

Describe the Feature

For Databricks Compatibility, add the following:

1. Buildkite testing:
   • Update pre-command (example)
   • Update pipeline.yml (example)
   • Update sample.profiles.yml (example)
   • Add the below to integration_tests/dbt_project.yml if it's not there:

     
     dispatch:

   • macro_namespace: dbt_utils search_order: ['spark_utils', 'dbt_utils']

2. For source packages, update src yml so a database won't be passed to spark (example or use below):

   sources: 
   - name: <name>
   database: "{% if target.type != 'spark' %}{{ var('<name>_database', target.database) }}{% endif %}"

3. Update any incremental models to update partition_by for databricks and add current strategies if not present:

   config(
       materialized='incremental',
       unique_key='<original unique key>',
       partition_by={'field': '<original field>', 'data_type': '<original data type>'} if target.type not in ('spark','databricks') else ['<original field>'],
       incremental_strategy = 'merge' if target.type not in ('postgres', 'redshift') else 'delete+insert',
       file_format = 'delta' 
   )

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• [ ] Yes.
• [ ] Yes, but I will need assistance and will schedule time during your office hours for guidance.
• [ ] No.

Anything else?

No response