Feature/shopify api updates

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: Issue #60, Issue #67, and Issue #66

This PR will result in the following new package version: v0.9.0

This will be a breaking change due to the removal of fields from the staging models. Customers who previously relied on these fields will no longer be able to leverage them. This was done to be in line with the connector API updates for Shopify.

Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:

• In June 2023 the Shopify connector received an update which upgraded the connector to be compatible with the new 2023-04 Shopify API. As a result, the following fields have been removed as they were deprecated in the API upgrade:

model	field removed
stg_shopify__customer	lifetime_duration
stg_shopify__order_line	fulfillment_service
stg_shopify__order_line	destination_location_* fields
stg_shopify__order_line	origin_location_* fields
stg_shopify__order	total_price_usd
stg_shopify__order	processing_method

• Removed databricks from the shopify_database configuration in the src_shopify.yml to allow Databricks Unity catalog users to define a unity Catalog as a database.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• [X] dbt compile
• [X] dbt run –full-refresh
• [X] dbt run
• [X] dbt test
• [n/a] dbt run –vars (if applicable)

Before marking this PR as "ready for review" the following have been applied:

• [X] The appropriate issue has been linked and tagged
• [X] You are assigned to the corresponding issue and this PR
• [X] BuildKite integration tests are passing

Detailed Validation

Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":

• [X] You have validated these changes and assure this PR will address the respective Issue/Feature.
• [X] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
• [X] You have provided details below around the validation steps performed to gain confidence in these changes.

Not much validation was needed for these changes. I used the release notes as the basis for what fields were being deprecated. I then removed these fields from the staging models, macros, and yml documentation. Ran the models and confirmed the fields were no longer present. No further validation was necessary per my opinion.

Standard Updates

Please acknowledge that your PR contains the following standard updates:

• Package versioning has been appropriately indexed in the following locations:
  • [X] indexed within dbt_project.yml
  • [X] indexed within integration_tests/dbt_project.yml
• [X] CHANGELOG has individual entries for each respective change in this PR
• [X] README updates have been applied (if applicable)
• [n/a] DECISIONLOG updates have been updated (if applicable)
• [X] Appropriate yml documentation has been added (if applicable)

dbt Docs

Please acknowledge that after the above were all completed the below were applied to your branch:

• [X] docs were regenerated (unless this PR does not include any code or yml updates)

If you had to summarize this PR in an emoji, which would it be?

🛫

[fivetran-reneeli]

Hi @fivetran-joemarkiewicz just reviewed, please see the below:

1. I'm wondering if not all deprecated fields were removed. For example, I noticed fulfillment_service is still in product_variant, though that field in product_variant object is no longer provided in their API. Do we need to remove it in that corresponding model and its ymls as well? I mean, I do still see it in the schema that was provided, but that schema also still has fulfillment_service in order_line, which was removed.

2. Still on topic of product_variant, I see that this issue mentioned that title also needs to be removed.

3. The changelog mentions shopify_using_shop_metafields added to the readme but I'm not seeing it? Nor do I see a Adding Metafields section

[fivetran-joemarkiewicz]

Hey @fivetran-reneeli thanks for reviewing, see my responses below:

I'm wondering if not all deprecated fields were removed. For example, I noticed fulfillment_service is still in product_variant, though that field in product_variant object is no longer provided in their API. Do we need to remove it in that corresponding model and its ymls as well?

• fulfillment_service was not deprecated from the product_variant table. Per the connector updates, that field was removed from a few tables, but not the product_variant table.
• Regarding the deprecated fields still being seen in the schema, this will be the case for connectors that were active before the upgrade. The connector isn't dropping these fields, but is instead just not syncing new records anymore. The fields still may exist, but we just won't sync new records anymore. However, for new connectors, these fields will not exist.
• Regarding the deprecated fields still being seen in the schema, this will be the case for connectors that were active before the upgrade. The connector isn't dropping these fields, but is instead just not syncing new records anymore. The fields still may exist, but we just won't sync new records anymore. However, for new connectors, these fields will not exist.

Still on topic of product_variant, I see that this https://github.com/fivetran/dbt_shopify/issues/60 mentioned that title also needs to be removed.

• Yup, this was originally assumed to be deprecated. However, per the notes from the PM, this field is still possible to be synced from a different endpoint. Therefore, it is still being synced and not being deprecated. I can update the issue description.

The changelog mentions shopify_using_shop_metafields added to the readme but I'm not seeing it? Nor do I see a Adding Metafields section

• Good catch, this is something that only should exist in the transform package. Removed from the PR.

[fivetran-reneeli]

Thanks for clarifying Joe! Just had one more comment ^

[fivetran-joemarkiewicz]

Thanks @fivetran-reneeli! Just updated and marked for re-review.

[fivetran-reneeli]

You might need to regen docs again but other than that, looks good pending buildkite 👍