dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://cloud.google.com/dataform/docs
Apache License 2.0
851 stars 163 forks source link

Dry run fails when table/view/assertion to be created does not exists #1805

Open ashish10alex opened 3 months ago

ashish10alex commented 3 months ago

There are two possible errors relating to dry run:

Tables / views

  1. If the table created by a .sqlx file is not already materialised on BigQuery the dry run will fail due to dry run trying to setMetadata on an view/table that does not exist yet. Relevant lines of code

https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/commands/run.ts#L371

https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/dbadapters/bigquery.ts#L235-L239

In these lines metadata for the table/view that already does not exist will be undefined and trying to access .schema attribute of undefined results in the error shown below

CleanShot 2024-08-12 at 19 16 41@2x


Assertion

Assertion dry run will face if the table it is trying to create does not already exists. Possibly due to trying to run the following query in the current code base which tries to select from a object that does not exist yet

https://github.com/dataform-co/dataform/blob/54dfa9ce51b141124608f0b5e15fa01f714e093a/cli/api/dbadapters/execution_sql.ts#L177


Minimal example to reproduce

  1. Add the following as a new .sqlx file to a dataform project
    
    config {
    type: "table",
    schema: "dataset_name",
    description: "any non empty description",
    tags: ["TEST_DRY_RUN"]
    }

SELECT 1 AS A UNION ALL SELECT 1 AS B


To recreate this error for view/assertion type, the type can be replaced by either of the two

2. Run  
   ```bash
   dataform run --dry-run --tags="TEST_DRY_RUN"

dataform version: 3.0.2 and also tested by running it from source

St-Hakky commented 1 week ago

Hi, I'm dataform user.

I’m also encountering the same issue, so I wanted to check if there have been any updates on this.

Ekrekr commented 1 week ago

Agreed, we shouldn't be attempting to set metadata on a dry run!

Any chance you can submit a patch yourself? https://github.com/dataform-co/dataform/blob/main/contributing.md