microsoft / dbt-fabric

MIT License
77 stars 26 forks source link

Committed changes created by dbt fail to update with Fabric workspace git. #231

Open timdenouden opened 1 month ago

timdenouden commented 1 month ago

Description

When an existing dbt project is run against a Fabric warehouse in a git enabled workspace. Any models using the ref() macro will fail to update in a separate Warehouse when attempting to update via git. This appears to be due to the three part naming for models that exist in the Warehouse being unable to be found by the Fabric workspace git merge system. I am unsure if this issue is a problem with the Fabric git merge system or with this library issuing the 3 part names for intra Warehouse references (I expect this depends on intended functionality in Fabric).

We have found a workaround by overriding the ref macro with:

{# macros/ref.sql #}

{% macro ref(model_name) %}

    {% do return(builtins.ref(model_name).include(database=false)) %}

{% endmacro %}

This works by removing the database name (The name of the Warehouse) which appears to confuse the Warehouse project git merge.  

Steps to recreate:

  1. Create a new empty warehouse (A) artifact in a Fabric workspace connected to git.
  2. Target and run a dbt project with views that reference other entities within the warehouse.
  3. Commit the new Warehouse changes to your branch in Workspace A
  4. In a separate Workspace (B), link the same branch and attempt to update from the branch based on Workspace A
  5. An error will appear saying the view with the references cannot find the source objects as they don't exist.
dataders commented 4 weeks ago

yo @timdenouden -- what's good!

I haven't looked too much into the issue here, just wanted to call out that there's still a gap in suport within SQL for three-part names for some commands. In those cases it's required to execute a USE {DATABSE}; statement first. Maybe something like that is going on?

@prdpsvs can share more here, but check out #161 and #166 for examples.