tconbeer / sqlfmt

sqlfmt formats your dbt SQL files so you don't have to
https://sqlfmt.com
Apache License 2.0
390 stars 16 forks source link

Unquoted strings in Jinja macro leads to bad formatting #498

Closed benjamin-awd closed 10 months ago

benjamin-awd commented 10 months ago

Describe the bug Unquoted strings within dbt Jinja macros are parsed improperly, and lead to unexpected/breaking changes

To Reproduce Run sqlfmt on a macro

{% macro generate_external_location(lake_zone, prefix) -%}
    s3://data-lake-{{ lake_zone }}-{{ target.name }}/{{ prefix }}/foo/bar
{%- endmacro %}

Expected behavior No changes are made

Actual behavior On sqlfmt.com, strings are split, leading to a bad S3 URI.

{% macro generate_external_location(lake_zone, prefix) -%}
    s3: / / data - lake -{{ lake_zone }}-{{ target.name }}/{{ prefix }}/ foo / bar
{%- endmacro %}

For some reason, I am getting a different (but still breaking) result locally

{% macro generate_external_location(lake_zone, prefix) -%}
    s3:  -- data-lake-{{ lake_zone }}-{{ target.name }}/{{ prefix }}/foo/bar
{%- endmacro %}

Additional context sqlfmt, version 0.20.0

In both cases, it seems that the tokens are being read literally, i.e. // is read as a SQL comment, and is transformed into --.

The result of the generate_external_location macro feeds into a dbt-athena config block, like so:

{{
    config(
        materialized='incremental',
        incremental_strategy='insert_overwrite',
        external_location=generate_external_location("bucket", "object=foo"),
        partitioned_by=['batch_date']
    )
}}
tconbeer commented 10 months ago

Macros can template anything, but we assume they are templating sql. An S3 URI is not valid SQL. Just place single quotes around it or --fmt: off on a line above the URI.

In snowflake // is a comment marker; your second example is from a more recent version of sqlfmt, where we standardize that comment marker to --