search

fivetran / dbt_netsuite

Data models for Fivetran's Netsuite connector, built using dbt.
https://fivetran.github.io/dbt_netsuite/
Apache License 2.0
36 stars 35 forks source link

[Bug] dbt source snapshot-freshness erroneously checks netsuite source tables as well as netsuite2 tables when only netsuite2 is supposed to be modeled #57

Closed wwilliamsnuwest closed 1 year ago

wwilliamsnuwest commented 1 year ago

Is there an existing issue for this?

Describe the issue

I have a check on source freshness running and am using the netsuite2 bit of the dbt netsuite package. However, the source freshness check (dbt source snapshot-freshness command) is checking freshness for BOTH the netsuite as well as the netsuite2 sources, and therefore returning errors for all the (stale/non-existent) netsuite source tables. Instead, I want source freshness to just check netsuite2 tables.

How can I turn off the netsuite checking? If this configurable, or is this something that has to be fixed in the dbt netsuite package code itself?

Thanks! Ward

Relevant error log or model output

02:01:19  2 of 72 START freshness of netsuite.accounting_periods.......................... [RUN]
02:01:19  3 of 72 START freshness of netsuite.accounts.................................... [RUN]
02:01:19  4 of 72 START freshness of netsuite.classes..................................... [RUN]
02:01:21  1 of 72 ERROR freshness of netsuite.accounting_books............................ [ERROR in 1.92s]
02:01:21  5 of 72 START freshness of netsuite.consolidated_exchange_rates................. [RUN]
02:01:21  3 of 72 ERROR freshness of netsuite.accounts.................................... [ERROR in 1.94s]
02:01:21  6 of 72 START freshness of netsuite.currencies.................................. [RUN]
02:01:21  2 of 72 ERROR freshness of netsuite.accounting_periods.......................... [ERROR in 1.97s]
02:01:21  7 of 72 START freshness of netsuite.customers................................... [RUN]
02:01:21  4 of 72 ERROR freshness of netsuite.classes..................................... [ERROR in 1.99s]
02:01:21  8 of 72 START freshness of netsuite.departments................................. [RUN]
02:01:22  5 of 72 ERROR freshness of netsuite.consolidated_exchange_rates................. [ERROR in 1.92s]
02:01:22  9 of 72 START freshness of netsuite.expense_accounts............................ [RUN]
02:01:22  7 of 72 ERROR freshness of netsuite.customers................................... [ERROR in 1.88s]
02:01:22  10 of 72 START freshness of netsuite.income_accounts............................ [RUN]
02:01:23  8 of 72 ERROR freshness of netsuite.departments................................. [ERROR in 1.93s]
02:01:23  11 of 72 START freshness of netsuite.items...................................... [RUN]
02:01:23  6 of 72 ERROR freshness of netsuite.currencies.................................. [ERROR in 2.08s]
02:01:23  12 of 72 START freshness of netsuite.locations.................................. [RUN]
02:01:24  9 of 72 ERROR freshness of netsuite.expense_accounts............................ [ERROR in 1.65s]
02:01:24  13 of 72 START freshness of netsuite.subsidiaries............................... [RUN]
02:01:24  11 of 72 ERROR freshness of netsuite.items...................................... [ERROR in 1.60s]
02:01:24  14 of 72 START freshness of netsuite.transaction_lines.......................... [RUN]
02:01:24  12 of 72 ERROR freshness of netsuite.locations.................................. [ERROR in 1.53s]
02:01:24  15 of 72 START freshness of netsuite.transactions............................... [RUN]
02:01:24  10 of 72 ERROR freshness of netsuite.income_accounts............................ [ERROR in 1.71s]
02:01:24  16 of 72 START freshness of netsuite.vendor_types............................... [RUN]
02:01:25  13 of 72 ERROR freshness of netsuite.subsidiaries............................... [ERROR in 1.37s]
02:01:25  17 of 72 START freshness of netsuite.vendors.................................... [RUN]
02:01:26  16 of 72 ERROR freshness of netsuite.vendor_types............................... [ERROR in 1.95s]
02:01:26  18 of 72 START freshness of netsuite2.account................................... [RUN]
02:01:26  14 of 72 ERROR freshness of netsuite.transaction_lines.......................... [ERROR in 2.02s]
02:01:26  19 of 72 START freshness of netsuite2.account_type.............................. [RUN]
02:01:26  15 of 72 ERROR freshness of netsuite.transactions............................... [ERROR in 2.25s]
02:01:27  20 of 72 START freshness of netsuite2.accounting_book........................... [RUN]
02:01:27  17 of 72 ERROR freshness of netsuite.vendors.................................... [ERROR in 1.37s]
02:01:27  21 of 72 START freshness of netsuite2.accounting_book_subsidiaries.............. [RUN]
02:01:29  20 of 72 PASS freshness of netsuite2.accounting_book............................ [PASS in 1.98s]
02:01:29  22 of 72 START freshness of netsuite2.accounting_period......................... [RUN]
02:01:29  19 of 72 PASS freshness of netsuite2.account_type............................... [PASS in 2.42s]

Expected behavior

I want source freshness to just check netsuite2 tables since in my dbt_project.yml file I've set 'netsuite_data_model: netsuite2'.

vars: netsuite_database: 'pc_fivetran_db' netsuite_schema: 'netsuite2_suiteanalytics' netsuite_data_model: netsuite2

My packages.yml file has this:

packages:

dbt Project configurations

vars: netsuite_database: 'pc_fivetran_db' netsuite_schema: 'netsuite2_suiteanalytics' netsuite_data_model: netsuite2 netsuite: transaction_lines_pass_through_columns:

Package versions

packages:

What database are you using dbt with?

snowflake

dbt Version

1.0

Additional Context

No response

Are you willing to open a PR to help address this issue?

wwilliamsnuwest commented 1 year ago

@Csgoodman FYI in case you're in a position to help!

wwilliamsnuwest commented 1 year ago

Ok, I found a way to just disable the source tables to avoid the freshness check that is in the netsuite package. I put this block of code in my dbt_project.yml file, and that seemed to do the trick (in both dev and prod.)

The issue raised still feels like a bug since I would think I wouldn't have to explicitly disabled the 'netsuite' stuff when I am clearly selecting just the 'netsuite2' data model in my dbt_project.yml file already, but maybe I'm just confused or am doing something else wrong?

sources:
  netsuite:
    netsuite:
      +enabled: false
  netsuite_source:
    netsuite:
      +enabled: false
fivetran-sheringuyen commented 1 year ago

Hey @wwilliamsnuwest! I am inclined to agree that this does seem like a bug in the package and agree with you that if you have declared netsuite_schema: 'netsuite2_suiteanalytics' then the package should only run freshness on just one source are the other. I am adding this into the queue to get looked into closer by our team at the next available sprint. Thank you for bringing this to our attention!

wwilliamsnuwest commented 1 year ago

Awesome; thanks for investigating further Sheri!

From: Sheri Nguyen @.> Sent: Monday, December 12, 2022 7:51 PM To: fivetran/dbt_netsuite @.> Cc: Ward Williams @.>; Mention @.> Subject: Re: [fivetran/dbt_netsuite] [Bug] dbt source snapshot-freshness erroneously checks netsuite source tables as well as netsuite2 tables when only netsuite2 is supposed to be modeled (Issue #57)

Hey @wwilliamsnuwesthttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwwilliamsnuwest&data=05%7C01%7Cwwilliams%40nuwestgroup.com%7C705b7f603c2b4dfec74f08dadcbd5bb7%7C5f334365592d48068c5dc237622d6bfe%7C0%7C0%7C638065003135455939%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MJFFVsT7C0FZLnNFeD%2FV47k52cdHWsX8Qs5Sq6lZm6s%3D&reserved=0! I am inclined to agree that this does seem like a bug in the package and agree with you that if you have declared netsuite_schema: 'netsuite2_suiteanalytics' then the package should only run freshness on just one source are the other. I am adding this into the queue to get looked into closer by our team at the next available sprint. Thank you for bringing this to our attention!

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffivetran%2Fdbt_netsuite%2Fissues%2F57%23issuecomment-1347705777&data=05%7C01%7Cwwilliams%40nuwestgroup.com%7C705b7f603c2b4dfec74f08dadcbd5bb7%7C5f334365592d48068c5dc237622d6bfe%7C0%7C0%7C638065003135455939%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2TBJZ4Fpoy6CN4qJCnz3Z1biaqrY3MgCJeXPM4csick%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAYA77U6EE55Y2IZCDHWNMJLWM7XFFANCNFSM6AAAAAASYTRECY&data=05%7C01%7Cwwilliams%40nuwestgroup.com%7C705b7f603c2b4dfec74f08dadcbd5bb7%7C5f334365592d48068c5dc237622d6bfe%7C0%7C0%7C638065003135455939%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LXlPmkcTNf%2FmzUQtKGg%2Be3ln3cccDRPuIwIvSu3eY%2B4%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

fivetran-jamie commented 1 year ago

hi there @wwilliamsnuwest i think this may be occurring due to conflicting dbt-core versions. so we disable netsuite vs netsuite2 source tables dynamically (based on netsuite_data_model) using the enabled config. this property was rolled out for sources in dbt v1.1.0 and it sounds like y'all are on v1.0.0? could you possibly update and try it out again? thanks!

fivetran-jamie commented 1 year ago

^ just tested this out myself in a virtual environment.

in dbt v1.0.0, all netsuite source tables are freshness-tested regardless of the value of netsuite_data_model. when i upgraded dbt to v1.1.0, this no longer occurred, and source freshness tests ran dynamically. so i think the solution here would be to upgrade your dbt version to at least v1.0.0

wwilliamsnuwest commented 1 year ago

Awesome; thanks Jamie!

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Jamie Rodriguez @.> Sent: Tuesday, December 27, 2022, 10:49 AM To: fivetran/dbt_netsuite @.> Cc: Ward Williams @.>; Mention @.> Subject: Re: [fivetran/dbt_netsuite] [Bug] dbt source snapshot-freshness erroneously checks netsuite source tables as well as netsuite2 tables when only netsuite2 is supposed to be modeled (Issue #57)

^ just tested this out myself in a virtual environment.

in dbt v1.0.0, all netsuite source tables are freshness-tested regardless of the value of netsuite_data_model. when i upgraded dbt to v1.1.0, this no longer occurred, and source freshness tests ran dynamically. so i think the solution here would be to upgrade your dbt version to at least v1.0.0

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffivetran%2Fdbt_netsuite%2Fissues%2F57%23issuecomment-1366103564&data=05%7C01%7Cwwilliams%40nuwestgroup.com%7Cafa6c6b0ca35497ffb0a08dae83b071b%7C5f334365592d48068c5dc237622d6bfe%7C0%7C0%7C638077637487183882%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5wZ1rFqT9UGrzPOKHAWDGSXdaIvA43xEmy8jfAqVluE%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAYA77U2NQVCT3OOMHQSXRRTWPM22BANCNFSM6AAAAAASYTRECY&data=05%7C01%7Cwwilliams%40nuwestgroup.com%7Cafa6c6b0ca35497ffb0a08dae83b071b%7C5f334365592d48068c5dc237622d6bfe%7C0%7C0%7C638077637487340090%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CtEnyPstGmjV2UeC1aC3BoE6Bp265%2FHlUVzoOfxvmDU%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

fivetran-jamie commented 1 year ago

sweet will close this out then! but if the issue persists for some reason feel free to open it back up 🤠