search

fivetran / dbt_netsuite

Data models for Fivetran's Netsuite connector, built using dbt.
https://fivetran.github.io/dbt_netsuite/
Apache License 2.0
36 stars 34 forks source link

[Feature] Adding incremental functionality to the balance sheet #100

Closed fivetran-avinash closed 4 months ago

fivetran-avinash commented 8 months ago

Is there an existing feature request for this?

Describe the Feature

The Netsuite Balance Sheet was designed to be cumulative and capture historical transactions, which is why those records are held in each subsequent accounting period. This produces the most value for the end user as it is easy for end users to tie out values to the Netsuite Balance Sheet UI.

However, over time, adding historical records to each accounting period increases the row count significantly (think Fibonacci sequence, where the table is growing even more widely in scale). This is leading to performance issues where tables are loading millions of historical rows that rarely change.

Adding incremental functionality so that only the most recent rows from recent accounting periods are loaded into the balance sheet would improve performance and lead to a better customer experience, while also providing the same longstanding functionality that the initial balance sheet build was designed for..

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

Anything else?

No response

jmongerlyra commented 8 months ago

I am interested in the approach taken on this enhancement once it is decided.

fivetran-catfritz commented 4 months ago

Hi @jmongerlyra I've made a test branch with incremental end models. If you'd like to test it out, below is the branch name:

  - git: https://github.com/fivetran/dbt_netsuite.git
    revision: feature/performance-improvements
    warn-unpinned: false

For Snowflake, I aimed to optimize for a shorter runtime since my understanding is that Snowflake charges based on time as opposed to data scanned, but let me know if you have any thoughts on that!

fivetran-catfritz commented 4 months ago

@jmongerlyra This update has been released! You can install it with the below range. 

packages:
  - package: fivetran/netsuite
    version: [">=0.13.0", "<0.14.0"]

I am closing this issue, but please ping us here if you have any additional comments or questions.