Closed dbrtly closed 8 months ago
Also I tried overriding the model (and insert_resources_from_graph) in my project but now I get a ZeroDivisionError in file "".
Hi. Could you please share the details of the SQL error on the BQ side?
Yes, the model is very unconventional but this is because we use some very unconventional ways to load the graph as well 😄
Does stg_exposures ever actually hydrated with data? How?
https://github.com/search?q=repo%3Adbt-labs%2Fdbt-project-evaluator%20stg_exposures&type=code
Why is there extra yml configuration in these unit tests compared to the models directory of the main project?
Yes, those models are ingesting data in an unconventional way. We iterated quite a bit around this to solve different performance issues etc...
Today, those staging tables are hydrated/loaded via the post_hook
in the config
{{
config(
materialized='table',
post_hook="{{ insert_resources_from_graph(this, resource_type='exposures') }}"
)
}}
What this post_hook
does is that it generates the data in Jinja, and then loops through it doing a few insert into
statements. And for the post hook to work we create this weird empty table at first.
One of the disadvantage of this approach is that the compiled/run code doesn't show the SQL run as part of the hook (which is likely to be what fails in your use case)
Could you either:
By experience it might be related to some fields that have single quotes in them, and in that case we'd need to update the code to escape those single quotes.
Especially, can you check if there is a '
in one of the exposures owner name?
No single quotes inside the string but we do have values like these patterns:
---
owner:
name: 'Fred Flintstone'
email: 'fred@rocksinthehead.com'
---
owner:
name: Peter Parker
email: ''
---
owner:
name: "Fred Flintstone"
email: ''fred@rocksinthehead.com"
---
owner:
name: "Fred Flintstone"
email: ""
---
owner:
name: "Fred Flintstone"
email: ""
Is this one as-is in your project? email: ''fred@rocksinthehead.com"
There is a mix of single and double quotes here which might raise the issue.
I have pushed some changes to the package to the branch fix/exposure-owner-name
. You could try to install the package via git (instead of the Hub) and point it to the new branch to see if this fixes the error.
If it does, we can get it merged soon.
sorry it's taken me so long to get back to you. I can't find the branch
Thanks, let's assume that the fix we did was good and we can open another issue if it is not the case
Describe the bug
The query is weird. When I add this package to my dbt project and run it, this model fails. Suggest refactoring to conventionalize the query by moving the placeholders to the cte and selecting * from the cte. On my testing, this change makes the error go away.
Steps to reproduce
run the package on bigquery
Expected results
pass all
Actual results
4 of 71 OK created sql table model main.stg_exposures ........................ [FAIL in 0.04s]
Screenshots and log output
System information
unchanged from main
Which database are you using dbt with?
The output of
dbt --version
:Additional context
proposed change to models/staging/graph/stg_exposures.sql
Are you interested in contributing the fix?
yes. https://github.com/dbrtly/dbt-project-evaluator/tree/refactor-simplify-cte