TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.
https://sqlmesh.com
Apache License 2.0
1.82k stars 160 forks source link

STAR macro not working with Snowflake dialect #3381

Open mpcarter opened 6 hours ago

mpcarter commented 6 hours ago

When using the @STAR macro with a snowflake model, it mismatches the case of the quoted alias used for the table (relation) and those used in the expansion of the columns.

MODEL (
    name sqlmesh_example.star_model
);

SELECT
@STAR(sqlmesh_example.incremental_model)
from sqlmesh_example.incremental_model

Using snowflake dialect it render as

SELECT
  CAST("incremental_model"."ID" AS INT) AS "ID",
  CAST("incremental_model"."ITEM_ID" AS INT) AS "ITEM_ID",
  CAST("incremental_model"."EVENT_DATE" AS DATE) AS "EVENT_DATE"
FROM "SQLMESH"."SQLMESH_EXAMPLE"."INCREMENTAL_MODEL" AS "INCREMENTAL_MODEL"

The table alias is "INCREMENTAL_MODEL", but the table alias used with the columns is lowercase and quoted, "incremental_model". So it fails to deploy.

Using duckdb it renders as

SELECT
  CAST("incremental_model"."id" AS INT) AS "id",
  CAST("incremental_model"."item_id" AS INT) AS "item_id",
  CAST("incremental_model"."event_date" AS DATE) AS "event_date"
FROM "sqlmesh"."sqlmesh_example"."incremental_model" AS "incremental_model"
mpcarter commented 6 hours ago

I believe I found a fix. In the star macro, the table_identifier needs to be normalized.

    excluded_names = {
        normalize_identifiers(excluded, dialect=evaluator.dialect).name
        for excluded in exclude.expressions
    }
    quoted = quote_identifiers.this
    table_identifier = normalize_identifiers(alias or relation, dialect=evaluator.dialect).name # use this instead
    # table_identifier = alias.name or relation.name

    columns_to_types = {
        k: v for k, v in evaluator.columns_to_types(relation).items() if k not in excluded_names
    }

I copied the code for @STAR into a user-defined macro to test it and it rendered my example like this for snowflake dialect

SELECT
  CAST("INCREMENTAL_MODEL"."ID" AS INT) AS "ID",
  CAST("INCREMENTAL_MODEL"."ITEM_ID" AS INT) AS "ITEM_ID",
  CAST("INCREMENTAL_MODEL"."EVENT_DATE" AS DATE) AS "EVENT_DATE"
FROM "SQLMESH"."SQLMESH_EXAMPLE"."INCREMENTAL_MODEL" AS "INCREMENTAL_MODEL"