search

fivetran / dbt_zendesk_source

Fivetran's Zendesk Support source dbt package
https://fivetran.github.io/dbt_zendesk_source/#!/overview
Apache License 2.0
13 stars 18 forks source link

Custom Field Definition Not Working #45

Closed jimmiemunyi closed 7 months ago

jimmiemunyi commented 7 months ago

Is there an existing issue for this?

Describe the issue

Question on this, say I have a custom field like so:

Field Name: Foo Field ID: 12345

How should I structure it in vars?

I tried this and it failed:

dbt_project.yml

vars: zendeskticket_passthrough_columns: [Foo] I got the message: Unrecognized name: Foo at [415:9] compiled Code at zendesk_source/models/stg_zendeskticket.sql

Relevant error log or model output

Unrecognized name: Foo at [415:9]
compiled Code at zendesk_source/models/stg_zendesk__ticket.sql

Expected behavior

The custom field to be picked up by dbt models

dbt Project configurations

Project name -- the same with Github Repo name

name: 'data_dbt' version: '1.3.0' config-version: 2

These configurations specify where dbt should look for different types of files.

model-paths: ["models"] analysis-paths: ["analysis"] test-paths: ["tests"] seed-paths: ["seeds"] macro-paths: ["macros"] snapshot-paths: ["snapshots"]

target-path: "target" # directory which will store compiled SQL files clean-targets: # directories to be removed by dbt clean

vars: quickbooks_schema: landing_quickbooks stripe_schema: landing_stripe stripe__using_credit_notes: True zendesk_schema: landing_zendesk using_domain_names: False using_organization_tags: False

zendesk__ticket_passthrough_columns: [Inquiry]

Package versions

packages:

What database are you using dbt with?

bigquery

dbt Version

1.3.0

Additional Context

None

Are you willing to open a PR to help address this issue?

fivetran-reneeli commented 7 months ago

Hi @jimmiemunyi ! I'd be happy to take a look.

I tried testing with some sample data using the same branch you're on and it seems the configs work as expected and allow stg_zendesk__ticket to run successfully.

Curious what warehouse you are using? Have you tried adding single or double quotes? And is it case sensitive (you're sure the column is Foo and not FOO or foo?

vars:
    zendesk__ticket_passthrough_columns: [custom_field]
jimmiemunyi commented 7 months ago

Yes, tried again with the double quotes and still not working:

Errors parsed from dbt logs:
BigQuery adapter: https://console.cloud.google.com/bigquery?project=data-pipeline&page=queryresults
1 of 3 ERROR creating sql table model staging_zendesk.stg_zendesk__ticket ...... [ERROR in 2.12s]
  Database Error in model stg_zendesk__ticket (models/stg_zendesk__ticket.sql)
  Unrecognized name: Inquiry at [415:9]
  compiled Code at target/data_dbt_dbt_assets/run/zendesk_source/models/stg_zendesk__ticket.sql

We are using Google BQ as our Warehouse

jimmiemunyi commented 7 months ago

For the name, I am getting it from: landing_zendesk.ticket_custom_field and using the title column from there

fivetran-reneeli commented 7 months ago

@jimmiemunyi the next question I have is if you've confirmed that Inquiry exists in your raw ticket table?

jimmiemunyi commented 7 months ago

Maybe that could be my issue. Which table specifically are you talking about?

It exists on the custom_fields table I mentioned above.

fivetran-reneeli commented 7 months ago

Ahh. So how the passthrough feature works is that it is enabled in our staging model for the raw ticket table . So the fields you want to add must come from that source table, the raw zendesk ticket table. Hopefully that helps clarify.

If you want to add in a custom field from a separate table, you could potentially try joining that table in on top of our models. But this is up to the user's judgement since this is outside of our package offering and you'll have to make certain with the keys you are joining on.

fivetran-reneeli commented 7 months ago

Hi @jimmiemunyi let me know if you have any other questions! Otherwise we'll probably close this one out in the coming days

jimmiemunyi commented 7 months ago

Hi @fivetran-reneeli, we can close this. Thanks for your guidance. I managed to get it to work