search

fivetran / dbt_shopify

Fivetran's Shopify dbt package
https://fivetran.github.io/dbt_shopify/
Apache License 2.0
52 stars 40 forks source link

Feature/performance updates #76

Closed fivetran-catfritz closed 4 months ago

fivetran-catfritz commented 5 months ago

PR Overview

This PR will address the following Issue/Feature:

This PR will result in the following new package version:

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

🚨 Breaking Changes 🚨

⚠️ Since the following changes are breaking, a --full-refresh after upgrading will be required.

  • Performance improvements:
  • Added an incremental strategy for of the following models. These models were picked for incremental materialization based on the size of their upstream sources. For Databricks destinations, the incremental updates have only been tested for compatibility with Databricks Runtime.
  • shopify__customer_cohorts
  • shopify__customer_email_cohorts
  • shopify__discounts
  • shopify__order_lines
  • shopify__orders
  • shopify__transactions
  • Updated the materialization of shopify__orders__order_line_aggregates to a table. This model draws on several large upstream sources and is also referenced in several downstream models, so this was done to improve performance. This was not selected for incremental materialization due to the nature of the aggregates used.

Features

  • Added a default 7-day look-back to incremental models to accommodate late arriving records. The number of days can be changed by setting the var lookback_window in your dbt_project.yml. See the Lookback Window section of the README for more details.

  • Updated the partitioning logic in window functions to use only the necessary columns, depending on whether the unioning feature is used. This benefits mainly Redshift destinations, which can see errors when the staging models are materialized as views.

Under the Hood

  • Updated the maintainer PR template to the current format.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

Before marking this PR as "ready for review" the following have been applied:

Detailed Validation

Please share any and all of your validation steps:

If you had to summarize this PR in an emoji, which would it be?

:dancer:
fivetran-catfritz commented 5 months ago

@fivetran-joemarkiewicz This is ready for re-review. I also made additional documentation and test updates that I realized were missing.