search

fivetran / dbt_shopify

Fivetran's Shopify dbt package
https://fivetran.github.io/dbt_shopify/
Apache License 2.0
52 stars 40 forks source link

[Bug] Compilation error in model int_shopify__order__shipping_aggregates #56

Closed kristof-taveirne closed 1 year ago

kristof-taveirne commented 1 year ago

Is there an existing issue for this?

Describe the issue

When trying to upgrade from the fivetran/shopify dbt package version 0.7 to 0.8 I'm experiencing the following Compilation Errors in the models.

However when I updated from DBT-bigquery 1.4.0 to 1.4.5 the error disappeared.

This probably means that the require-dbt-version needs to be updated from [">=1.3.0", "<2.0.0"] to [">=1.4.5", "<2.0.0"] for the version 0.8.0 to work.

Kind regards, Kristof.

Relevant error log or model output

`Compilation Error in model int_shopify__order__shipping_aggregates (models/intermediate/int_shopify__order__shipping_aggregates.sql)
13:27:09    dbt was unable to infer all dependencies for the model "int_shopify__order__shipping_aggregates".
13:27:09    This typically happens when ref() is placed within a conditional block.
13:27:09    
13:27:09    To fix this, add the following hint to the top of the model "int_shopify__order__shipping_aggregates":
13:27:09    
13:27:09    -- depends_on: {{ ref('stg_shopify__order_shipping_line') }}
13:27:17  20 of 24 OK created sql table model dev_kristof_base.shopify__order_lines ...... [CREATE TABLE (4.8m rows, 2.3 GB processed) in 8.45s]
13:27:17  22 of 24 SKIP relation dev_kristof_base.shopify__products due to ephemeral model error  [ERROR SKIP]
13:27:17  Compilation Error in model int_shopify__products_with_aggregates (models/intermediate/int_shopify__products_with_aggregates.sql)
13:27:17    dbt was unable to infer all dependencies for the model "int_shopify__products_with_aggregates".
13:27:17    This typically happens when ref() is placed within a conditional block.
13:27:17    
13:27:17    To fix this, add the following hint to the top of the model "int_shopify__products_with_aggregates":
13:27:17    
13:27:17    -- depends_on: {{ ref('stg_shopify__collection_product') }}
13:27:17  23 of 24 SKIP relation dev_kristof_entities.products due to ephemeral model error  [ERROR SKIP]
13:27:17  Compilation Error in model int_shopify__products_with_aggregates (models/intermediate/int_shopify__products_with_aggregates.sql)
13:27:17    dbt was unable to infer all dependencies for the model "int_shopify__products_with_aggregates".
13:27:17    This typically happens when ref() is placed within a conditional block.
13:27:17    
13:27:17    To fix this, add the following hint to the top of the model "int_shopify__products_with_aggregates":
13:27:17    
13:27:17    -- depends_on: {{ ref('stg_shopify__collection_product') }}
13:27:17  24 of 24 SKIP relation dev_kristof_bi.product_assortment_analysis due to ephemeral model error  [ERROR SKIP]
13:27:17  Compilation Error in model int_shopify__products_with_aggregates (models/intermediate/int_shopify__products_with_aggregates.sql)
13:27:17    dbt was unable to infer all dependencies for the model "int_shopify__products_with_aggregates".
13:27:17    This typically happens when ref() is placed within a conditional block.
13:27:17    
13:27:17    To fix this, add the following hint to the top of the model "int_shopify__products_with_aggregates":
13:27:17    
13:27:17    -- depends_on: {{ ref('stg_shopify__collection_product') }}
13:27:17  
13:27:17  Finished running 9 view models, 15 table models in 0 hours 0 minutes and 24.21 seconds (24.21s).
13:27:17  
13:27:17  Completed with 4 errors and 0 warnings:
13:27:17  
13:27:17  Compilation Error in model.shopify.shopify__orders, caused by compilation error in referenced ephemeral model model.shopify.int_shopify__order__shipping_aggregates
13:27:17  
13:27:17  Compilation Error in model.shopify.shopify__products, caused by compilation error in referenced ephemeral model model.shopify.int_shopify__products_with_aggregates
13:27:17  `

Expected behavior

These models come fresh out of the box so I would expect those models to generate without an issue.

dbt Project configurations

vars: shopify_union_schemas: [ 'shopify_store_nl', 'shopify_store_fr', 'shopify_store_en', ] # use this if the data is in different schemas/datasets of the same database/project shopify_database: 'pitpit-data-raw' shopify_using_fulfillment_event: false

Package versions

packages:

What database are you using dbt with?

bigquery

dbt Version

Core:

Plugins:

Additional Context

No response

Are you willing to open a PR to help address this issue?

fivetran-joemarkiewicz commented 1 year ago

HI @kristof-taveirne thanks so much for opening this issue and raising it to our attention.

This is a peculiar bug indeed 🤔. I just downgraded my dbt version to 1.4.0 and found that it compiled and ran successfully. I also checked the dbt-core and dbt-bigquery releases and did not see any regression that called this type of error message out.

Further, I took a look at the error messages and they seem to be contradicting the logic within the models themselves. The error messages state that this may be caused by having refs live within conditional blocks; however, none of those models have the refs in conditional blocks. In theory, dbt should be able to pick these referential dependencies up without an issue.

I am a bit apprehensive of adjusting the dbt required range as this can cause some friction for our existing users that are not on the latest version of dbt-core. I wonder if this could have possibly been a result of needing to clean the manifest and when upgrading the package there somehow was conflicting manifest versions of the package that caused this error to occur?

fivetran-joemarkiewicz commented 1 year ago

As mentioned above, this is not something we will likely adjust within the package. That being said, I would be open to hearing if any other users run into this same issue.

However, for the time being I will close this issue.