Closed fivetran-catfritz closed 3 months ago
This PR will address the following Issue/Feature:
This PR will result in the following new package version:
Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:
Performance Improvements Updated the sequence of JSON parsing for model fivetran_platform__audit_table to reduce runtime. Bug Fixes Updated model fivetran_platform__audit_user_activity to correct the JSON parsing used to determine column email. Under the hood Updated logic for macro fivetran_log_lookback to align with logic used in similar macros in other packages.
fivetran_platform__audit_table
fivetran_platform__audit_user_activity
email
fivetran_log_lookback
Please acknowledge that you have successfully performed the following commands locally:
Before marking this PR as "ready for review" the following have been applied:
Please share any and all of your validation steps:
See internal ticket for data quality validation
Runtime comparison:
log
where cast(created_at as date) < cast('2024-05-10' as date) and cast(created_at as date) >= cast('2024-05-07' as date)
which produces these results:
v1.7.2 run: ~ 1hr
v1.7.3 run: ~ 10 mins
PR Overview
This PR will address the following Issue/Feature:
124
This PR will result in the following new package version:
Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:
PR Checklist
Basic Validation
Please acknowledge that you have successfully performed the following commands locally:
Before marking this PR as "ready for review" the following have been applied:
Detailed Validation
Please share any and all of your validation steps:
See internal ticket for data quality validation
Runtime comparison:
log
data limited to about 250k rows. This was done by filtering the staging log model using filterwhich produces these results:
v1.7.2 run: ~ 1hr
v1.7.3 run: ~ 10 mins
If you had to summarize this PR in an emoji, which would it be?
🌳