brooklyn-data / dbt_artifacts

A dbt package for modelling dbt metadata. https://brooklyn-data.github.io/dbt_artifacts

Apache License 2.0

331 stars 126 forks source link

[Fix] Add batching when uploading models #344

Closed jared-rimmer closed 1 year ago

jared-rimmer commented 1 year ago

Overview

In release 2.4.0 was added the all_results JSON object to upload_models.sql
This results in large queries that are too big for BigQuery which we saw for tests.
This PR uses the exact same logic for tests but in models.

Update type - breaking / non-breaking

[ ] Minor bug fix
[ ] Documentation improvements
[ ] Quality of Life improvements
[X] New features (non-breaking change)
[ ] New features (breaking change)
[ ] Other (non-breaking change)
[ ] Other (breaking change)

What does this solve?

343

Outstanding questions

Need confirmation on what the correct batch size should be

What databases have you tested with?

[ ] Snowflake
[X] Google BigQuery
[ ] Databricks
[ ] Spark
[ ] N/A