bugfix/too-many-partitions

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: Internally raised issue

This PR will result in the following new package version: v0.8.0

This will be adjusting the partition granularity for all incremental models. This should only impact BigQuery users. However, it will still result in the need for a full refresh. Therefore, this should be a breaking change.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes (Full refresh required after upgrading)

• Removed the partition_by logic from incremental models running on BigQuery. This change affects only BigQuery warehouses and resolves the too many partitions error that some users encountered. The partitioning was also deemed unnecessary for the mentioned models and their downstream references, offering no performance benefit. By removing it, we eliminate both the error risk and an unneeded configuration. This change applies to the following models:
  • int_klaviyo__event_attribution
  • klaviyo__events

Under the Hood

• Added consistency and integrity validation tests for the klaviyo__events model.
• Cleaned up unnecessary variable configuration within the integration_tests/dbt_project.yml file.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• [X] dbt run –full-refresh && dbt test
• [X] dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

• [X] The appropriate issue has been linked, tagged, and properly assigned
• [X] All necessary documentation and version upgrades have been applied
• [X] docs were regenerated (unless this PR does not include any code or yml updates)
• [x] BuildKite integration tests are passing
• [x] Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

See below for the results of the validation tests (full refresh and incremental runs)

If you had to summarize this PR in an emoji, which would it be?

📅

[fivetran-joemarkiewicz]

@fivetran-catfritz thanks for this initial review. I agree that the partition logic was not necessary and added more complexity and issues than was actually helping here since the partitioned field was not being utilized in the downstream or incremental logic. Therefore, the latest update removes this logic. I have also re-ran the validtation tests and can see the results below.

Let me know if there are any other comments or questions.