dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://cloud.google.com/dataform/docs
Apache License 2.0
850 stars 162 forks source link

Override database config is not applied to Auto generated assertions #1295

Open newuniverse opened 2 years ago

newuniverse commented 2 years ago

About this https://docs.dataform.co/guides/assertions#auto-generated-assertions .

I am using dataform run --vars option to embed a new database(GCP project_id) into each sqlx files to override the default_database in my CI/CD.

Not from code base but from behavior base, I found that override is not applied to auto generated assertions.

Is it an expected behavior or just a bug?

For example:

dataform.json

{
    "warehouse": "bigquery",
    "defaultSchema": "dataform",
    "assertionSchema": "dataform_assertions",
    "defaultDatabase": "my-dev-project"
}

some_query.sqlx

config {
  type: "table",
  database: dataform.projectConfig.vars.embed_my_prd_database,
  assertions: {
    uniqueKey: ["user_id"],
    nonNull: ["user_id", "customer_id"],
    rowConditions: [
      'signup_date is null or signup_date > "2019-01-01"',
      'email like "%@%.%"'
    ]
  }
}
select ...

If you run dataform run --actions=some_query --vars=embed_my_prd_database=my-prd-project

expect: success

but I got followings because I was using SA credential of my-prd-project not of my-dev-project

Assertion failed:  dataform_assertions.some_query_assertions_uniqueKey_0
  > 
  >       create or replace view `my-dev-project.dataform_assertions.some_query_assertions_uniqueKey_0` as
Ekrekr commented 1 year ago

A more elegant solution that we have planned for this is CLI flags for dataform.json parameters.

newuniverse commented 1 year ago

I'm looking forward to it!

Ekrekr commented 11 months ago

Config option was not the fix, https://github.com/dataform-co/dataform/pull/1602 demonstrates however confirms that this was not an issue with Core's dataform.json interpretation.

Will create a new PR to add these to the tests, and fix the underlying cause.

Ekrekr commented 11 months ago

I'm a bit stumped by this, but I've narrowed down the issue and it is clear it only happens in the CLI.

Adding more tests for Dataform Core (https://github.com/dataform-co/dataform/pull/1603) demonstrates that:

Adding more tests for Dataform CLI (https://github.com/dataform-co/dataform/pull/1604) demonstrates that:

btelFD commented 7 months ago

Also running into this. We set the database in the config block of each .sqlx and assertions in the config block don't work