[Bug] Assumes presence of optional tables

[jayhale]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

The package assumes the presence of three tables from Fivetran that may not be synced:

• account_membership
• destination_membership
• user

These tables are not created by the Fivetran Log connector, and are not filtered in the schema configuration for the connector.

Relevant error log or model output

16:28:55  Database Error in model stg_fivetran_log__account_membership (models\staging\stg_fivetran_log__account_membership.sql)
16:28:55    002003 (42S02): SQL compilation error:
16:28:55    Object 'PC_FIVETRAN_DB.FIVETRAN_LOG.ACCOUNT_MEMBERSHIP' does not exist or not authorized.
16:28:55    compiled SQL at target\run\fivetran_log\models\staging\stg_fivetran_log__account_membership.sql
16:28:55
16:28:55  Database Error in model stg_fivetran_log__destination_membership (models\staging\stg_fivetran_log__destination_membership.sql)
16:28:55    002003 (42S02): SQL compilation error:
16:28:55    Object 'PC_FIVETRAN_DB.FIVETRAN_LOG.DESTINATION_MEMBERSHIP' does not exist or not authorized.
16:28:55    compiled SQL at target\run\fivetran_log\models\staging\stg_fivetran_log__destination_membership.sql
16:28:55
16:28:55  Database Error in model stg_fivetran_log__user (models\staging\stg_fivetran_log__user.sql)
16:28:55    002003 (42S02): SQL compilation error:
16:28:55    Object 'PC_FIVETRAN_DB.FIVETRAN_LOG.USER' does not exist or not authorized.
16:28:55    compiled SQL at target\run\fivetran_log\models\staging\stg_fivetran_log__user.sql
16:28:55
16:28:55  Done. PASS=270 WARN=0 ERROR=3 SKIP=4 TOTAL=277

Expected behavior

Gracefully handle missing tables or allow the user to disable tables that are not present.

dbt Project configurations

# dbt_project.yml

# ...

vars:
  # fivetran/fivetran_log package configuration
  fivetran_log_database: pc_fivetran_db
  fivetran_log_schema: fivetran_log
  fivetran_log_using_transformations: false
  fivetran_log_using_triggers: false

Package versions

# packages.yml

packages:
  - package: fivetran/fivetran_log
    version: [">=0.6.0", "<0.7.0"]

What database are you using dbt with?

snowflake

dbt Version

dbt --version
Core:
  - installed: 1.1.0
  - latest:    1.1.0 - Up to date!

Plugins:
  - snowflake: 1.1.0 - Up to date!

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [X] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @jayhale thanks so much for opening this issue. You are correct that the package expects these tables to be syncing in your connector. They should in fact be syncing within the connector itself. The one idea I have is if you may be on the free trial period of the connector?

If not, we will want to explore PR #52 further to allow users to disable these models if they are not being synced by default. Let me know about the above and we can explore this further. Thanks!

[jayhale]

@fivetran-joemarkiewicz we are well past the free trial period.

[fivetran-joemarkiewicz]

Hi @jayhale thanks for the clarification! My team and I will prioritize the PR I mentioned above to ensure you will be able to leverage the package effectively without seeing the error.

However, in the meantime you actually will be able to disable these models without needing to make any changes to the package. You can leverage the below configuration within your dbt_project.yml to disable the models from being run during your model executions.

models:
  fivetran_log:
    staging:
      stg_fivetran_log__account_membership:
        +enabled: false
      stg_fivetran_log__destination_membership:
        +enabled: false
      stg_fivetran_log__user:
        +enabled: false

let me know if adding this removes the error!

[fivetran-joemarkiewicz]

In addition to the model config showed above, the latest 0.6.1 release of the package incorporates variables to allow you to disable the source tables if you do not have them in your warehouse.

Thanks again @jayhale for raising this to our team! I will close this issue as it has been addressed in the latest release.