Bug/redshift json parse

[fivetran-catfritz]

PR Overview

This PR will address the following Issue/Feature:

• 113

• Ticket T-621124

This PR will result in the following new package version:

• v1.5.0: I made this breaking mostly since I changed the partitioning for bigquery and cluster for snowflake. It won't cause errors, but the warehouses won't change the partition unless there's a full refresh. Also I was feeling cautious since we are changing the logic of the json_parse macro and also adding the lookback for the incremental fivetran_platform__audit_table. Though tested, these are updates that could introduce unexpected behavior.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes

• The following changes are marked as a breaking change out of caution, as a full refresh may be required if you are experiencing issues after the update.
• For Bigquery and Databricks users, updated the partition_by config to coordinate with the filter used in the incremental logic.
• For Snowflake users, added a cluster_by config for performance.

Feature Updates

• Updated incremental logic for fivetran_platform__audit_table so that it looks back 7 days to catch any late arriving records.
• Updated json parsing logic to prevent run failures when incoming json-like strings are invalid.
• Added filter to fivetran_platform__connector_status so only necessary log records will be parsed.

Under The Hood

• Added macros:
  • fivetran_log_json_parse to handle the updated json parsing.
  • fivetran_log_lookback for use in fivetran_platform__audit_table.
• Updated testing of invalid json strings.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• [x] dbt run –full-refresh && dbt test
• [x] dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

• [x] The appropriate issue has been linked, tagged, and properly assigned
• [x] All necessary documentation and version upgrades have been applied
• [x] docs were regenerated (unless this PR does not include any code or yml updates)
• [x] BuildKite integration tests are passing
• [x] Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

• See validation worksheet

If you had to summarize this PR in an emoji, which would it be?

🔍