[Bug] dbt manifest generation failed

[trueclassic02]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

I'm attempting to set up the ad reporting model package in Fivetran, but I'm encountering issues getting it to execute. While it works smoothly in my local environment, the same error persists when I try to run it in Fivetran.

Relevant error log or model output

The node semantic_model.ad_reporting.ad_report not found

Expected behavior

Create ad reporting models and schedule them in fivetran transformations screen

dbt Project configurations

name: 'fivetran_ad_reports'
version: '1.0.0'
config-version: 2

profile: 'fivetran_ad_reports'

vars:
  amazon_ads_schema: amz_ads_schema
  amazon_ads_database: ad_db
  facebook_ads_schema: facebookads_schema
  facebook_ads_database: ad_db
  google_ads_schema: googleads_schema
  google_ads_database: ad_db

  microsoft_ads_schema: bingads_schema
  microsoft_ads_database: ad_db
  pinterest_schema: pinterestads_schema
  pinterest_database: ad_db
  tiktok_ads_schema: tiktokads_schema
  tiktok_ads_database: ad_db

  ad_reporting__apple_search_ads_enabled: False
  ad_reporting__linkedin_ads_enabled: False
  ad_reporting__reddit_ads_enabled: False
  ad_reporting__snapchat_ads_enabled: False
  ad_reporting__twitter_ads_enabled: False

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

clean-targets:    
  - "target"
  - "dbt_packages"

models:  
  ad_reporting:
    +schema:

  amazon_search_ads:
    +schema:
  amazon_ads_source:
    +schema:

  facebook_ads:
    +schema:
  facebook_ads_source:
    +schema:

  google_ads:
    +schema:
  google_ads_source:
    +schema:

  microsoft_ads:
    +schema:
  microsoft_ads_source:
    +schema:

  pinterest:
    +schema:
  pinterest_source:
    +schema:

  tiktok_ads:
    +schema:
  tiktok_ads_source:
    +schema:

Package versions

packages:

• package: fivetran/ad_reporting version: 1.7.0

What database are you using dbt with?

bigquery

dbt Version

1.7.3

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [X] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @trueclassic02 thanks for opening this issue. This does seem to be similar to Issue #106; however, I am still unsure what could possibly be causing the error you are seeing. Are you by chance doing any customer Semantic layer modeling? I wonder if that could be conflicting with the semantic model provided in the package.

@Jstein77 it seems this is another case where we are getting a similar type of Semantic Layer error only when deploying to an orchestration tool. It works just fine locally, but not in Fivetran Transformations with dbt Core, or in dbt Cloud as seen in the other ticket. Any ideas?

[Jstein77]

@fivetran-joemarkiewicz Hmm I'm not sure, so we're seeing the issue when trying to run dbt core in Fivetran? What version of dbt core is running in Fivetran? Happy to hop on a call to try and debug live.

[fivetran-joemarkiewicz]

Good catch @Jstein77! If the user is running dbt v1.7.3 locally that would be reasonable why it is succeeding, but then fail with an error like this if a dbt version >v1.6.0 is being used in the orchestration layer.

[fivetran-joemarkiewicz]

I just tried to recreate this error locally, and while I do not get the exact same error I do see a similar type error when trying to run using dbt v1.3.7.

@trueclassic02 would you be able to share what version of dbt you are using within Fivetran Transformations? If it is less than dbt v1.6.0 then this may be the reason for the error you are seeing as the latest version of the package requires a version greater than or equal to v1.6.0

[trueclassic02]

@fivetran-joemarkiewicz We have set it to dbtcore version - 1.7.3 and we are not doing any custom semantic layer modeling as well. It'll be great if we can schedule some time to get on a call and debug this as @Jstein77 suggested.

[fivetran-joemarkiewicz]

@trueclassic02 this seems to possibly be branching into the realm of more an orchestration question than a package question. It likely may be an issue with what Fivetran Transformations is performing during the initial manifest generation as I was able to see that the issue is not actually occurring during the run of a model, but just during the connection of the Git repo.

Therefore, I have reached out to some folks internally who are on the Fivetran Transformations team to setup a call with you to better troubleshoot this error. I'll be sure to still join the call, but the other folks involved with the orchestration layer will likely be more help. Please let me know if you don't see someone from the Fivetran team reach out to you in the next day or so. I look forward to chatting soon!

[fivetran-joemarkiewicz]

Upon investigating this further, we found that the root cause of this issue is due to the Fivetran Transformations orchestration layer is not properly generating the semantic_manifest.json. This is why the models succeed as expected when run locally, but fail in deployment on Fivetran Transformations.

As such, there is no update that will be applied to the dbt package code. Instead, an update will need to be made to the Fivetran Transformations orchestration. Therefore, I will mark this ticket as wontfix and you can follow along in the Fivetran support ticket to receive more information and updates regarding any changes to Fivetran Transformations in order for the issue you are seeing to be resolved.

Thanks again for opening this issue and working with us to understand the root cause.