search

autotraderuk / dbt-dry-run

Dry run capability for dbt projects using BigQuery
https://pypi.org/project/dbt-dry-run/
Apache License 2.0
85 stars 12 forks source link

Unrecognized name _PARTITIONTIME for models with time ingestion partitioning #73

Closed malikfm closed 2 months ago

malikfm commented 3 months ago

I have two models, Model A (upstream) and B (downstream). Model A is an incremental model and it sets "time_ingestion_partitioning": true in its config. Model B selects from Model A using partition filter, e.g. WHERE _PARTITIONTIME = 'YYYY-MM-DD HH:MM:SS'.

dbt dry run fails on Model B with below error:

Node model.dbt_project.model_b failed with exception:
400 POST https://bigquery.googleapis.com/bigquery/v2/projects/gcp-project/jobs?prettyPrint=false: Unrecognized name: _PARTITIONTIME
ccharlesgb commented 3 months ago

This is because the dry runner does not add the _PARTITIONTIME column to the expected schema of Model A so it doesn't get included in the select literal.

Probably not that hard to fix, the snapshot_runner does something similar. We just need to say if the dry run is successful and time_ingestion_partitioning == true then add the _PARTITIONTIME column to the predicted schema. It would have to be just after this line I think

You would need to pull in the ingestion time config into the Node when the manifest is parsed as well