BUG - Table names in src_klaviyo.yml aren't configurable

[tokybe]

Are you a current Fivetran customer? Hi, Toky here, I'm a data engineer at Dija.

Describe the bug Since the table names in src_klaviyo.yml are hard-coded and not using the following variables

vars:
  klaviyo_source:
    event_table: "{{ source('klaviyo', 'event') }}" 
    campaign: "{{ source('klaviyo', 'campaign') }}" 
    flow: "{{ source('klaviyo', 'flow') }}" 
    integration: "{{ source('klaviyo', 'integration') }}" 
    person: "{{ source('klaviyo', 'person') }}" 
    metric: "{{ source('klaviyo', 'metric') }}"

dbt source snapshot-freshness will always look for the default table names.

Steps to reproduce

1. Set different target names for the klaviyo_source tables
2. Run dbt source snapshot-freshness
3. It fails with

   Runtime Error in source campaign (models/src_klaviyo.yml)
   404 Not found: Table <project_id>:<klaviyo_source_schema>.campaign was not found in location <location>

Expected behavior dbt source snapshot-freshness finds klaviyo source tables when they don't have their default names.

Project variables configuration

models:
 klaviyo_source:
     +schema:  # leave blank for just the target_schema

vars:
  klaviyo_source:
    event_table: "{{ ref('stg_klaviyo__event_base') }}" 
    campaign: "{{ ref('stg_klaviyo__campaign_base') }}" 
    flow: "{{ ref('stg_klaviyo__flow_base') }}" 
    integration: "{{ ref('stg_klaviyo__integration_base') }}" 
    person: "{{ ref('stg_klaviyo__person_base') }}" 
    metric: "{{ ref('stg_klaviyo__metric_base') }}"
  klaviyo_schema:

Package Version

packages:
  - package: fivetran/klaviyo_source 
    version: [">=0.1.0","<0.2.0"]

Warehouse

- [X] BigQuery - [ ] Redshift - [ ] Snowflake - [ ] Postgres - [ ] Databricks - [ ] Other (provide details below) **Additional context** **Screenshots**

Please indicate the level of urgency This doens't allow us to have freshness information for our klaviyo source tables.

Are you interested in contributing to this package?

• [ ] Yes, I can do this and open a PR for your review.
• [ ] Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• [X] No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[fivetran-joemarkiewicz]

Hi @tokybe thanks for opening this issue!

I have a few quick clarifying questions before we dig into this a bit more. The main issue is that your Klaviyo source tables are named differently than the hard coded source table names we have defined within our variables? I see the error you are receiving is around the campaign source table not existing. What is the name of your campaign table? My understanding is that the Fivetran schema produces default table names, is this not the case?

[tokybe]

Thanks for the quick reply!

The klaviyo_source package allows the Klaviyo source tables to be named differently thanks to the project's variable configuration, so our campaign table is called stg_klaviyo__campain_base. stg_klaviyo__campaign_tmp.sql works fine because it's selecting from {{ var('campaign') }} but the src_klaviyo.yml has campaign hard coded. It should also point to {{ var('campaign') }} instead.

I hope that clarifies the issue.

[fivetran-joemarkiewicz]

@tokybe I think I am misunderstanding the request. The klaviyo_source package allows you to point the sources to different sources but not rename it. In order for the freshness tests to work you will need to point the variable to a relevant source (ie. "{{ source('klaviyo', 'campaign') }}" and not a reference). Is the stg_klaviyo__campagin_base the name of the table that Fivetran is syncing?

I would be happy to jump on a quick call during our dbt office hours with you to discuss further with you if needed.

[tokybe]

Sorry, I meant point to different sources, not rename them. I am pointing the variables to relevant sources but the freshness tests doesn't use the variables. That's the issue. I've booked a slot on your calendly for a quick call.

[fivetran-jamie]

hey @tokybe -- i believe the solution would be to create a new src_klaviyo.yml file in your project's models folder that overrides the klaviyo_source package sources. you can find the formatting to do so here.

to retain all of the column descriptions and such, i think you would basically need to copy and paste all of the contents of the package's src_klaviyo.yml, add the overrides: klaviyo_source attribute, and change the problematic table names (ie campaign -> campaign_base). happy to provide more context or try to hop on a call again if that doesn't work or make sense!

[tokybe]

Hi @fivetran-jamie, ok, I'll do that. Thanks!