[Bug] discount_id not present

[bgeils]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

I am attempting to run the Shopify dbt pipeline and get errors due to a missing field in a source table. The missing field is discount_id from the source table: abandoned_checkout_discount_code

It is an expected input column noted here -- and necessary to successfully run the discount pipeline path. https://github.com/fivetran/dbt_shopify_source/blob/main/macros/get_abandoned_checkout_discount_code_columns.sql

It is also missing from the ERD documentation: https://docs.google.com/presentation/d/1wSWI7SbY4NMtyRLWdg2Z4LW3-SRCF8K7McN0VzLjh3w/edit#slide=id.g1ba00f96b0c_2_0

Screenshot of missing column in FT schema.

Relevant error log or model output

No response

Expected behavior

The field to exist in the schema/database and subsequently the data to be ingested.

dbt Project configurations

NA

Package versions

packages:

• package: fivetran/shopify version: [">=0.8.0", "<0.9.0"]

What database are you using dbt with?

bigquery

dbt Version

NA (prior to dbt pipeline)

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [X] No.

[fivetran-jamie]

hey there! so it seems like discount_id is a deprecated field that we had in our own data, but hasn't been populated in a long time. we should definitely remove this field and use code instead for any tests in stg_shopify.yml (i think we already use code for joins downstream)

are you only getting testing related errors, or anything else?

[fivetran-jamie]

hey @bgeils i think i've addressed the problematic spots here

i have a working branch if you'd like to test it out! 🤞

packages:
  - git: https://github.com/fivetran/dbt_shopify_source.git
    revision: bugfix/abandoned-checkout-discount-id

[fivetran-jamie]

another customer with this issue just tested the above branch out and confirmed that the error was resolved on their end at least!

[fivetran-jamie]

actually it'd be great if you could test this out as well when ya have a chance @bgeils

[brian-equilife]

@fivetran-jamie I'm having this same issue and I'd like to try testing this fix. I'm not clear how to do that. I tried dropping the package info below into my packages.yml file but I still had the error compiling the discount table. I suspect it's because I'm using the fivetran/dbt_shopify package which includes and include for the existing release with the bug. [Sorry if I'm breaking some git protocol etiquette here - I'm new to this]

hey @bgeils i think i've addressed the problematic spots here

i have a working branch if you'd like to test it out! 🤞

packages:
  - git: https://github.com/fivetran/dbt_shopify_source.git
    revision: bugfix/abandoned-checkout-discount-id

[fivetran-jamie]

@brian-equilife hi there! yeah so you should comment out the the fivetran/dbt_shopify package in your packages.yml, as you're right, there'll be conflicting staging versions otherwise. so, replace the shopify lines with

  - git: https://github.com/fivetran/dbt_shopify_source.git
    revision: bugfix/abandoned-checkout-discount-id

and run dbt clean (if you have a clean-paths config in your dbt_project.yml, make sure it includes dbt_packages instead of dbt_modules) and then dbt run (note, this will only run the staging models, not the downstream end models).

and you're totally not breaking any sort of protocol, glad you jumped in!

[fivetran-catfritz]

@brian-equilife following up to Jamie's post, I created a branch of our transform package fivetran/dbt_shopify that points to the above staging branch if that's easier to test/confirm that the changes work downstream. You can use the below in your packages yml instead of the above:

packages:
  - git: https://github.com/fivetran/dbt_shopify.git
    revision: test/shopify_source_fix

Thank you!

[brian-equilife]

Thank you @fivetran-jamie and @fivetran-catfritz This fix worked like a charm and I no longer have any errors with the shopify__discounts table. I really appreciate your detailed instructions.

[fivetran-jamie]

this is in the latest release! if you're using the shopify transform package, it'll automatically pick up the changes in the source here