search

fivetran / dbt_netsuite

Data models for Fivetran's Netsuite connector, built using dbt.
https://fivetran.github.io/dbt_netsuite/
Apache License 2.0
36 stars 34 forks source link

[Feature] Allow us the ability to turn off using the TransactionAccountingLine table from Netsuite #77

Closed mikerenderco closed 1 year ago

mikerenderco commented 1 year ago

Is there an existing feature request for this?

Describe the Feature

The TransactionAccountingLine table from Netsuite is an extremely expensive table to sync through Fivetran and for all the modeling we do we don't use any fields from this table. It would be nice if we could NOT sync this table and the DBT job still run successfully.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

Anything else?

No response

fivetran-jamie commented 1 year ago

hi there @mikerenderco, thanks for taking the time to open this issue! Do you know what Netsuite release version you are one? I ask because we've explored this in the past and came away with the following:

A huge driver of the high MAR has been this table's inability to sync incrementally. There was no last_modified_timestamp column, so for each sync, every single row was resynced and recounted, even if nothing changed. This was due to a limitation on Netsuite's side that they have since resolved

The transaction_accounting_line table will officially have a last_modified_timestamp to it in the Netsuite 2023.1 release. This release has already begun for some customers, however each customer has their own rollout from the Oracle side. Our connector will immediately begin syncing incrementally for this table once a customers table appears with this field.

mikerenderco commented 1 year ago

Hi @fivetran-jamie - So I'm super confused. Our company is on 2023.1. I have opened many tickets with Fivetran support on this and I have attached their email string here. [Fivetran Support] Re Re MAR Question.pdf

I struggle with this response though because we have 8 connections in Fivetran - two of which have a netsuite account and we have to pulll the TransactionAccountingLine table for both. Of all the connections we have these two tables make up 98% of our costs. Even after the last_modified_timestamp has been added.

fivetran-jamie commented 1 year ago

Ah sorry to add to your confusion @mikerenderco, I wasn't aware of the issue with the checksum methodology. Fivetran is working on an alternative delete capture approach (no firm timeline though) BUT there is potentially a workaround for the time being....

We are currently working with another customer to suss this out further, but what you could do is, instead of deleting records in the Netsuite UI, fully zero them out. This way, they will not have an impact on downstream reports. However, I advise that you first check that zero-ing out a transaction does not trigger a deletion in any related tables. We don't know if this works fully yet, but could be worth a shot. I can also update you here with the other customer's success (or lack thereof) with this approach

As for the package, the TransactionAccountingLine table is unfortunately pretty crucial due it housing the amount fields 😢

mikerenderco commented 1 year ago

Ok no worries. You can close this issue. I was just trying to figure out how to control our costs. Looking forward to the delete capture approach! If that comes out it would be much appreciated.

On Wed, Aug 30, 2023 at 4:03 PM Jamie Rodriguez @.***> wrote:

Ah sorry to add to your confusion @mikerenderco https://github.com/mikerenderco, I wasn't aware of the issue with the checksum methodology. Fivetran is working on an alternative delete capture approach (no firm timeline though) BUT there is potentially a workaround for the time being....

We are currently working with another customer to suss this out further, but what you could do is, instead of deleting records in the Netsuite UI, fully zero them out. This way, they will not have an impact on downstream reports. However, I advise that you first check that zero-ing out a transaction does not trigger a deletion in any related tables. We don't know if this works fully yet, but could be worth a shot. I can also update you here with the other customer's success (or lack thereof) with this approach

As for the package, the TransactionAccountingLine table is unfortunately pretty crucial due it housing the amount fields https://github.com/fivetran/dbt_netsuite/blob/main/models/netsuite2/intermediate/base/int_netsuite2__transaction_lines.sql 😢

— Reply to this email directly, view it on GitHub https://github.com/fivetran/dbt_netsuite/issues/77#issuecomment-1699965294, or unsubscribe https://github.com/notifications/unsubscribe-auth/A23MTH5HHPBDZPUJ35GJ3CDXX7BFDANCNFSM6AAAAAA37BP6OE . You are receiving this because you were mentioned.Message ID: @.***>

-- Mike Lambeth 775.813.0944 @.***