fivetran / dbt_fivetran_log

Data models for Fivetran's internal log connector built using dbt.
https://fivetran.github.io/dbt_fivetran_log/
Apache License 2.0
30 stars 24 forks source link

bugfix/redshift-large-int #117

Closed fivetran-joemarkiewicz closed 5 months ago

fivetran-joemarkiewicz commented 6 months ago

PR Overview

This PR will address the following Issue/Feature: Internally raised issue

This PR will result in the following new package version: v1.6.0

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes

⚠️ Since the following changes result in a field changing datatype, we recommend running a --full-refresh after upgrading to this version to avoid possibly incremental failures.

  • The following fields in the fivetran_platform__audit_table model have been updated to be cast as dbt.type_bigint() (previously was dbt.type_int())
  • sum_rows_replaced_or_inserted
  • sum_rows_updated
  • sum_rows_deleted

Bug Fixes

Under the Hood

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

Before marking this PR as "ready for review" the following have been applied:

Detailed Validation

Please share any and all of your validation steps:

I was able to reproduce this error locally by editing the seed data to insert a count record in the message_data field within the log data to be a large integer that exists outside the normal range (Integer valid range -2147483648 to 2147483647). See the below screenshot where I was able to reproduce this error using the live version of the package.

image

Additionally, I was able to see this error be resolved when using this branch version of the package.

image

Lastly, we can see the resulting audit_table and connector_daily_events end model fields look as we would expect following these updates.

image

Following these validations, I feel comfortable this should address the issue being reported for the integers being out of range for the respective fields.

If you had to summarize this PR in an emoji, which would it be?

🔢