search

fivetran / dbt_netsuite

Data models for Fivetran's Netsuite connector, built using dbt.
https://fivetran.github.io/dbt_netsuite/
Apache License 2.0
32 stars 34 forks source link

[Bug] NETSUITE2__BALANCE_SHEET #99

Closed ryan-gillies closed 1 week ago

ryan-gillies commented 7 months ago

Is there an existing issue for this?

Describe the issue

We are getting an extraordinarily high row count from the netsuite2_balance_sheet model. For context, see the row counts of the other models we are using. The balance sheet model is causing severe performance issues due to its size. Can you help understand why the row count is so much higher and if this is expected to differ from the netsuite2_income_statement model so much? I expected them to be similar as both are pulling period end balances.

Relevant error log or model output

No response

Expected behavior

I expect that this model would not have more rows than total rows from transaction_details model

dbt Project configurations

https://fivetran.com/dashboard/transformations/quickstart/deservedly_rebelled?groupId=combatants_purest

Package versions

https://fivetran.com/dashboard/transformations/quickstart/deservedly_rebelled?groupId=combatants_purest

What database are you using dbt with?

snowflake

dbt Version

https://fivetran.com/dashboard/transformations/quickstart/deservedly_rebelled?groupId=combatants_purest

Additional Context

https://fivetran.com/dashboard/transformations/quickstart/deservedly_rebelled?groupId=combatants_purest

Are you willing to open a PR to help address this issue?

fivetran-avinash commented 7 months ago

Hi @ryan-gillies, thanks for opening this issue with us!

After investigating, it appears the balance sheet is working as intended. The way it's constructed, it was designed to be cumulative for each accounting period, gathering the current month's transactions and all historical transactions in each accounting period. You can see the joins for accounting_period_id in this model. so that these transactions are brought into each accounting period.

So in each subsequent accounting period, all previous historical transactions are being brought into the new accounting period. This is why the balance sheet model is growing significantly compared to the other models.

We do believe this approach provides the most end value for the customer, so we'd like to retain this approach, but we do understanding the increasing performance concerns of maintaining a growing balance sheet.

So we have opened a feature request to make this model incremental and bring in only the most recent data and reduce performance concerns. Encourage you to follow along there regarding development!

Let me know if this line of approach sounds good or if you have any questions. We will keep you in the loop about any further development on this issue.

fivetran-avinash commented 7 months ago

We will not be fixing this as this is expected behavior of the package. We will address the issue of model performance in the recently opened issue.