dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://cloud.google.com/dataform/docs
Apache License 2.0
846 stars 160 forks source link

uniqueKey assertions generate invalid SQL for incremental tables in v3.0.2 #1804

Open stevenleggdfe opened 2 months ago

stevenleggdfe commented 2 months ago

To reproduce, put the .sqlx file below in a Dataform repo with dataform-core v3.0.2 installed:

config {
  type: "incremental",
  assertions: {
    uniqueKey: ["foo"]
  }
}

SELECT 1 as foo, 2 AS bar

This generates the SQL below for the assertion (view):

SELECT
  *
FROM (
  SELECT
    TableAssertionsConfig,
    COUNT(1) AS index_row_count
  FROM `project-name.dataform.uniquekey_bug_test`
  GROUP BY TableAssertionsConfig
  ) AS data
WHERE index_row_count > 1

This in turn generates the compilation error "Unrecognized name: TableAssertionsConfig at [6:5]."

This bug does not occur when:

mattimoller commented 2 months ago

The same issue seems to be present for models of type view

blundin commented 2 months ago

I'm seeing this on models of type view as well.

Updated: This error occurs in v3.0.2 and v3.0.1. v3.0.0 does not throw this error.