search

brooklyn-data / dbt_artifacts

A dbt package for modelling dbt metadata. https://brooklyn-data.github.io/dbt_artifacts
Apache License 2.0
309 stars 113 forks source link

[Bug]: #395

Closed dougscc closed 9 months ago

dougscc commented 9 months ago

Overview

dbt_artifacts 2.6.0 fails when run under dbt 1.6.5 / dbt-bigquery 1.6.6 on BigQuery

How to reproduce

Install brookyn-data/dbt_artifacts 2.6.0 in conjunction with dbt-bigquery 1.6.6

Expected behaviour

dbt_artifacts runs normally

Screenshots

11:48:53  BigQuery adapter: https://console.cloud.google.com/bigquery?project=foo-bar-bat&j=bq:None:br549-tr6-5000-1-800-3mp1r3&page=queryresults
11:48:53  Database error while running on-run-end
11:48:53  
11:48:53
11:48:53  Finished running 137 view models, 600 tests, 3 seeds, 11 incremental models, 4 snapshots, 89 table models in 0 hours 6 minutes and 33.18 seconds (393.18s).
11:48:54  
11:48:54
11:48:54  Completed with 1 error and 3 warnings:
11:48:54  
11:48:54
11:48:54    on-run-end failed, error:
 The query is too large. The maximum standard SQL query length is 1024.00K characters, including comments and white space characters.

Environment

Results of running dbt --version:

$ dbt --version
Core:
  - installed: 1.6.5
  - latest:    1.6.5 - Up to date!

Plugins:
  - bigquery: 1.6.6 - Up to date!

Please paste the contents of your packages.yml file here:


packages:
  - package: brooklyn-data/dbt_artifacts
    version: ["<=2.6.0"]
glsdown commented 9 months ago

Hello. As noted in #369 we added a new feature in 2.6.0 which will hopefully help here. In a previous release, we added an additional all_results column which contained a JSON object of all results for people to be able to do their own analysis on. However, this increased the query size dramatically, which is why these issues started arising.

In the latest release, I've added a variable that will stop this behaviour to reduce the query size. Please can you try adding the following as a variable in your dbt_project.yml file and see if that resolves it?

vars:
  dbt_artifacts_exclude_all_results: true
dougscc commented 9 months ago

@glsdown Looks good! Thanks for the assist.