[Bug] Missing most source tables

jaredx435k2d0 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the issue

We have 1388 tables being synced by our NetSuite2 connector.

However, my stg_netsuite_suiteanalytics schema has only 40 views.

Relevant error log or model output

No response

Expected behavior

I expected to see all of those output in the schema I designated for this package in my dbt_project.yml:

  netsuite_source:
    +schema: stg_netsuite_suiteanalytics
    +materialized: view

I'd prefer to not have to replicate the remainder of them manually. How do I get the package to bring in all of my source tables?

dbt Project configurations

models:

  netsuite:
    +schema: netsuite_suiteanalytics
  netsuite_source:
    +schema: stg_netsuite_suiteanalytics
    +materialized: view

vars:

  # fivetran/netsuite package variables
  netsuite_database: [my_db_name_redacted]
  netsuite_schema: [my_schema_name_redacted]
  netsuite_data_model: netsuite2 #netsuite by default
  netsuite2__multibook_accounting_enabled: false
  netsuite2__using_vendor_categories: true

Package versions

packages:

package: dbt-labs/codegen version: 0.9.0
package: fivetran/zendesk version: [">=0.10.0", "<0.11.0"]
package: fivetran/netsuite version: [">=0.7.0", "<0.8.0"]

What database are you using dbt with?

snowflake

dbt Version

dbt --version
Core:
  - installed: 1.4.5
  - latest:    1.4.5 - Up to date!

Plugins:
  - snowflake: 1.4.2 - Up to date!

We are using Fivetran dbt Transformations. We have no deployment.yml file because we are using Scheduled with Fivetran. The dbt Core version we're using there is 1.3.2

Additional Context

No response

Are you willing to open a PR to help address this issue?

[ ] Yes.
[ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
[ ] No.

fivetran-joemarkiewicz commented 1 year ago

HI @jaredx435k2d0 thanks for reaching out with this issue!

This is a very relevant question; however, per our README we only provide these staging models out of the box with this package. We design these packages to be leveraged by the majority of our users. Therefore, we will only bring in the source tables that we are confident the most customers are syncing. Additionally, we typically follow our rule of only bringing in source tables that are used in our downstream transformations. While this is the case, we have on occasion brought in highly useful sources even if they are not used in the downstream transformations.

As you mentioned, your environment has 1388 tables. This is likely not the same setup for the majority of the other Netsuite customers. Since we want to ensure this package will work for the majority of users, we will likely not bring in these other staging models. I apologize that this package does not have full coverage over the source, but I hope you understand our decision around excluding the entirety of the source tables in this package.

Please let me know if you have any questions.

jaredx435k2d0 commented 1 year ago

Hey, Joe.

Understood! Thanks for the info.

What is the recommended best practice for other staging models a user may want? To have two staging schemas (one from netsuite_source and one that's created by the user)?

fivetran-joemarkiewicz commented 1 year ago

Of course! The best practice here would be to leverage the models that are already provided by the package, and then create your own staging models for the ones not covered by the package.

jaredx435k2d0 commented 1 year ago

Got it.

Do you happen to have any recommendations on how to best auto-generate those staging models without overwriting those created by the package? I currently use dbt-coves to auto-generate my staging models and sources. I'd be happy to switch to a different route, but I'm not aware of a method that'd allow me to put all my staging models in the same stg_netsuite_suiteanalytics schema without manual intervention each time to skip those that were created by the package.

Thoughts?

fivetran-joemarkiewicz commented 1 year ago

I have never used dbt-coves, but that seems like a pretty neat tool and I may need to look at leveraging it myself!

Based on the demo, I think it could work where you use dbt-coves to generate the models all the models and then just remove any duplicates that are found in the netsuite dbt package. It would require a bit of manual work, but would thankfully only be a one time operation.

jaredx435k2d0 commented 1 year ago

I see.

Given that there will be regular schema changes to the Fivetran-created schema (new tables, new columns, etc.), I need to run dbt-coves to update/regenerate sources/staging models on a regular basis.

Using this package, I'm not sure how someone would keep a staging schema up to date with the Fivetran-created schema without consistent manual intervention.

If anyone ends up having ideas, please share and tag me!

fivetran-joemarkiewicz commented 1 year ago

Will do!

Closing this issue for the time being, but open to re-opening if others have a similar request.

fivetran / dbt_netsuite_source