dbt-labs / dbt-bigquery

dbt-bigquery contains all of the code required to make dbt operate on a BigQuery database.
https://github.com/dbt-labs/dbt-bigquery
Apache License 2.0
215 stars 149 forks source link

[Bug] Cannot parse partition config when specified in property YAML file #1256

Closed ags-de closed 3 months ago

ags-de commented 3 months ago

Is this a new bug in dbt-bigquery?

Current Behavior

In my dbt project, I put all my config to .yml files specified to each model:

dbt_project/
  ├── models/
  │   ├── folder1/
  │   │   ├── model1.sql
  │   │   ├── model1.yml
...
-- model1.sql

select * from `source_bq_table`

{% if is_incremental() %}

where
  date(creation_time) > date_add(current_date(), interval -1 day)

{% endif %}

I have a target BigQuery incremental table partitioned by creation_time column. I based my config on offical dbt-bigquery documentation:

# model1.yml

version: 2

models:
  - name: model1
    config:
      materialized: incremental
      on_configuration_change: fail
      partition_by:
        - field: creation_time
        - data_type: timestamp
        - granularity: day

However, when I run the model, I get the Could not parse partition config error.

The only thing that seems to work is to put partition_by config to .sql file:

-- model1.sql

{{
  config(
    partition_by={
      "field": "creation_time",
      "data_type": "timestamp",
      "granularity": "day"
    },
 ) }}

select * from `source_bq_table`

{% if is_incremental() %}

where
  date(creation_time) > date_add(current_date(), interval -1 day)

{% endif %}

Expected Behavior

I want to be able to put all needed config in one place (partition_by in this case), specifically .yml file associated with the model.

Steps To Reproduce

With environment and file structure specified in Current Behavior, run: $ dbt run

Relevant log output

$ dbt run
08:40:34  Running with dbt=1.8.2
08:40:35  Registered adapter: bigquery=1.8.1
08:40:35  Unable to do partial parsing because saved manifest not found. Starting full parse.
08:40:36  Found 3 models, 469 macros
08:40:36  
08:40:37  Concurrency: 1 threads (target='dev')
08:40:37  
08:40:37  1 of 1 START sql incremental model dataset.model1  [RUN]
08:40:37  1 of 1 ERROR creating sql incremental model dataset.model1  [ERROR in 0.07s]
08:40:37  
08:40:37  Finished running 1 incremental model in 0 hours 0 minutes and 1.57 seconds (1.57s).
08:40:37  
08:40:37  Completed with 1 error and 0 warnings:
08:40:37
08:40:37    Runtime Error in model1 (models\folder1\model1.sql)
  Could not parse partition config
08:40:37
08:40:37  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Environment

- OS: Windows 10
- Python: 3.12.3
- dbt-core: 1.8.2
- dbt-bigquery: 1.8.1

Additional Context

dbt docs on BigQuery config image

source: https://docs.getdbt.com/reference/resource-configs/bigquery-configs

jtcohen6 commented 3 months ago

@ags-de The partition_by config is dictionary, not a list. Try this instead:

# model1.yml

version: 2

models:
  - name: model1
    config:
      materialized: incremental
      on_configuration_change: fail
      partition_by:
        field: creation_time
        data_type: timestamp
        granularity: day

And let us know if that works!

ags-de commented 3 months ago

@jtcohen6 thank you for swift response.

That did indeed work, I wouldn't thought about that. Much appreciated!