databricks / dbt-databricks

A dbt adapter for Databricks.
https://databricks.com
Apache License 2.0
226 stars 119 forks source link

SQL compilation error when running `--empty` flag on on model that utilizes `dbt_utils.union_relations()` macro #807

Open dbeatty10 opened 1 month ago

dbeatty10 commented 1 month ago

Describe the bug

I got an error when trying to use the --empty flag with a model that uses the dbt_utils.union_relations macro.

Workaround

Doing this update seemed to resolve the issue for me (just changing relation|lower to be relation.render()|lower):

{% macro get_columns_comments(relation) -%}
  {% call statement('get_columns_comments', fetch_result=True) -%}
    describe table {{ relation.render()|lower }}
  {% endcall %}

  {% do return(load_result('get_columns_comments').table) %}
{% endmacro %}

Steps To Reproduce

Create these files:

models/my_model.sql

select 1 as id

models/my_model_unioned.sql

with unioned as (
    {{ dbt_utils.union_relations(
        relations=[ref("my_model")]
    ) }}
)

select * from unioned

Run these commands:

dbt run -s +my_model_unioned --empty

Expected behavior

Completed successfully

Screenshots and log output

Actual log output:

$ dbt run -s +my_model_unioned --empty --profile databricks

16:47:05  Running with dbt=1.8.5
16:47:33  Registered adapter: databricks=1.8.5
16:47:33  Found 2 models, 708 macros
16:47:33  
16:47:36  Concurrency: 10 threads (target='databricks')
16:47:36  
16:47:36  1 of 2 START sql view model dbt_dbeatty.my_model ............................... [RUN]
16:47:38  1 of 2 OK created sql view model dbt_dbeatty.my_model .......................... [OK in 1.31s]
16:47:38  2 of 2 START sql view model dbt_dbeatty.my_model_unioned ....................... [RUN]
16:47:39  2 of 2 ERROR creating sql view model dbt_dbeatty.my_model_unioned .............. [ERROR in 0.97s]
16:47:40  
16:47:40  Finished running 2 view models in 0 hours 0 minutes and 6.54 seconds (6.54s).
16:47:40  
16:47:40  Completed with 1 error and 0 warnings:
16:47:40  
16:47:40    Runtime Error in model my_model_unioned (models/my_model_unioned.sql)
  Runtime Error

    [PARSE_SYNTAX_ERROR] Syntax error at or near '('. SQLSTATE: 42601 (line 2, pos 15)

    == SQL ==
    /* {"app": "dbt", "dbt_version": "1.8.5", "dbt_databricks_version": "1.8.5", "databricks_sql_connector_version": "3.1.2", "profile_name": "databricks", "target_name": "databricks", "node_id": "model.my_project.my_model_unioned"} */
    describe table (select * from `hive_metastore`.`dbt_dbeatty`.`my_model` where false limit 0) _dbt_limit_subq_my_model
    ---------------^^^

16:47:40  
16:47:40  Done. PASS=1 WARN=0 ERROR=1 SKIP=0 TOTAL=2

System information

The output of dbt --version:

$ dbt --version

Core:
  - installed: 1.8.5

Plugins:
  - databricks: 1.8.5 - Update available!

The operating system you're using: N/A

The output of python --version: N/A

Additional context

See https://github.com/dbt-labs/dbt-core/issues/10766#issuecomment-2374593273 for background.