dbt-labs / dbt-bigquery

dbt-bigquery contains all of the code required to make dbt operate on a BigQuery database.
https://github.com/dbt-labs/dbt-bigquery
Apache License 2.0
216 stars 152 forks source link

[Bug] Temporary table does not use partition expiry from incremental model configuration #1117

Open thomasdkelly opened 8 months ago

thomasdkelly commented 8 months ago

Is this a new bug in dbt-bigquery?

Current Behavior

In our GCP project, we set a default partition expiry of 3 years. However, we have one model that we want to go back to 5 years. This model has the following config:

{{
    config(
        alias = 'tom_demo',
        materialized = 'incremental',
        incremental_strategy = 'insert_overwrite',
        partition_by = {'field': 'date', 'data_type': 'date'},
        on_schema_change = 'append_new_columns',
        partition_expiration_days = 366 * 5
    )
}}

When attempting to backfill this model for 2021-01-01 (more than 3 years ago), the model is running but the resulting partition is empty. The query returns data but the temp table uses the default project partition expiry of 3 years (instead of the 5 years specified by the model).

Expected Behavior

I expect the temp table to have the same expiry of the model it wishes to merge into.

Steps To Reproduce

In your gcp project, attempt to run the model for a date within your models config partition expiry (but greater than your projects default expiry). For example this project has a default expiry of 3 years, which means this partition for 2021-01-01 will is outside of the temp table expiry. (but within the tom_demo model's expiry).

    create or replace table `project`.`intermediate`.`tom_demo__dbt_tmp`

    partition by snapshot_date

    OPTIONS(
      description="""creating temp table to show""",

      expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)
    )
    as (

      select 
      DATE("2021-01-01") as snapshot_date,
      1 as col
    )

Relevant log output

No response

Environment

- OS: linux
- Python: 3.11
- dbt-core: 1.6.6
- dbt-bigquery: 1.6.6

Additional Context

No response

ccharlesgb commented 2 months ago

@dbeatty10 Would you accept a PR for this? Just got hit by this again. We had a dataset that had a default partition expiration of 7 days. Couldn't understand why incremental model's with a partition expiration of 3 years were failing to insert any data.