search

fivetran / dbt_netsuite

Data models for Fivetran's Netsuite connector, built using dbt.
https://fivetran.github.io/dbt_netsuite/
Apache License 2.0
36 stars 35 forks source link

[Feature] High MAR from Transaction Accounting Line #49

Closed fivetran-jamie closed 1 year ago

fivetran-jamie commented 2 years ago

Is there an existing feature request for this?

Describe the Feature

A few customers have noted the transaction_accounting_line source table is a high MAR table and would prefer to disable this model if possible. This feature is intended to allow a variable that disables the transaction_accounting_line model without having an impact on the downstream Netsuite models.

Note: The transaction_accounting_line source table is a crucial component of our transaction line calculation. This is then used in all downstream financial report models. We will not be able to do a simple disable model variable as that would result in unusable downstream models. We will need to find another approach. See the "Anything else?" section of this FR for more details.

Describe alternatives you've considered

Not sync the source table in question and result in failures in the dbt run as the package requires this source.

Are you interested in contributing this feature?

Anything else?

Based on the dbt_netsuite DAG the stg_netsuite2__transaction_accounting_lines model is used in our downstream models. In fact, it is a crucial piece to capturing the total amounts for each transaction line. See our int_netsuite2__transaction_lines model where this model is used to generate the totals which are used downstream in the greater financial reports. image

Due to the importance of this source, I am curious how the package would be effective with its removal. A better alternative from disabling the source would be to find a way to allow for the totals of the transaction lines to be generated from somewhere else. However, that does not seem to be the case at the moment.

Also, this issue has a mirror issue in the source package to increase visibility https://github.com/fivetran/dbt_netsuite_source/issues/22

fivetran-jamie commented 1 year ago

Ok, so a huge driver of the high MAR was this table's inability to sync incrementally. There was no last_modified_timestamp column, so for each sync, every single row was resynced and recounted, even if nothing changed. This was due to a limitation on Netsuite's side that they have since resolved!! 🥳 🎉 🎈

The transaction_accounting_line table will officially have a last_modified_timestamp to it in the upcoming Netsuite 2023.1 release! This release has already begun for some customers, however each customer has their own rollout from the Oracle side. Our connector will immediately begin syncing incrementally for this table once a customers table appears with this field.

tldr; This table is significantly less problematic if you upgrade to Netsuite 2023.1