Feature/hubspotv3 api update new

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: Issue #111

This PR will result in the following new package version: v0.11.0

This will be a breaking change as the upstream dbt_hubspot_source package included major updates as a result of the API v3 changes (see our release notes here from the connector. As a result, many tables were impacted on how they sync data. As such, we should make this a breaking change and require customers to ensure they are on the latest version of the HubSpot connector in order to leverage this package.

I do also want to call out that I was able to make these changes compatible with the old version of the connector (pre v3 API update). That being said, when using the old version of the connector with this package, you will see some fields are not documented as they have been removed from the newer API version. Likewise, due to the backwards compatibility, there are some fields that will always be null for users of the new API. These fields have been identified in the yml documentation and notified for deprecation and removal once all connectors are upgraded (next few months).

Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:

🚨 Breaking Changes 🚨

This change is made breaking in part to updates applied to the upstream dbt_hubspot_source package following upgrades to ensure compatibility with the HubSpot v3 API updates. Please see below for the relevant upstream changes:

• Following the May 2023 connector update the HubSpot connector now syncs the below parent and child tables from the new v3 API. As a result the dependent fields and field names from the downstream staging models have changed depending on the fields available in your HubSpot data. Now the respective staging models will sync the required fields for the dbt_hubspot downstream transformations and all of your property_hs_* fields. Please be aware that the property_hs_* will be truncated from the field name in the staging and downstream models. The impacted sources (and relevant staging models) are below:

  - `ENGAGEMENT`
  - `ENGAGEMENT_CALL`
  - `ENGAGEMENT_COMPANY`
  - `ENGAGEMENT_CONTACT`
  - `ENGAGEMENT_DEAL`
  - `ENGAGEMENT_EMAIL`
  - `ENGAGEMENT_MEETING`
  - `ENGAGEMENT_NOTE`
  - `ENGAGEMENT_TASK`
  - `TICKET`
  - `TICKET_COMPANY`
  - `TICKET_CONTACT`
  - `TICKET_DEAL`
  - `TICKET_ENGAGEMENT`
  - `TICKET_PROPERTY_HISTORY`

  • Please note that while these changes are breaking, the package has been updated to ensure backwards compatibility with the pre HubSpot v3 API updates. As a result, you may see some null fields which are artifacts of the pre v3 API HubSpot version. Be sure to inspect the relevant field descriptions for an understanding of which fields remain for backwards compatibility purposes. These fields will be removed once all HubSpot connectors are upgraded to the v3 API.

• The engagements_joined macro has been adjusted to account for the HubSpot v3 API changes. In particular, the fields that used to only be available in the engagements source are now available in the individual engagement_[type] sources. As such, coalesce statements were added to ensure the correct populated fields are used following the join. Further, to avoid ambiguous columns a dbt-utils.star was added to remove the explicitly declared columns within the coalesce statements.

  • The coalesce is only included for backwards compatibility. This will be removed in a future release when all HubSpot connectors are on the latest API version.

Under the Hood

• The base_model argument used for the engagements_joined macro has been updated to be an explicit ref as opposed to the previously used var within the below models:
  • hubspot__engagement_calls
  • hubspot__engagement_emails
  • hubspot__engagement_meetings
  • hubspot__engagement_notes
  • hubspot__engagement_tasks
  • This update was applied as the var result was producing inconsistent results during compile and runtime when leveraging the dbt-utils.star macro. However, the explicit ref always provided consistent results.

Documentation Updates

• As new fields were added in the v3 API updates, and old fields were removed, the documentation was updated to reflect the v3 API consistent fields. Please take note if you are still using the pre v3 API, you will find the following end models no longer have complete field documentation coverage:
  • hubspot__engagement_calls
  • hubspot__engagement_emails
  • hubspot__engagement_meetings
  • hubspot__engagement_notes
  • hubspot__engagement_tasks

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• [X] dbt compile
• [X] dbt run –full-refresh
• [X] dbt run
• [X] dbt test
• [X] dbt run –vars (if applicable)

I ran various versions of the package with variables enabled and disabled. In particular I tested to ensure the changed models worked when enabled and disabled.

Before marking this PR as "ready for review" the following have been applied:

• [X] The appropriate issue has been linked and tagged
• [X] You are assigned to the corresponding issue and this PR
• [X] BuildKite integration tests are passing

Detailed Validation

Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":

• [X] You have validated these changes and assure this PR will address the respective Issue/Feature.
• [X] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
• [X] You have provided details below around the validation steps performed to gain confidence in these changes.

This package update has been validated to ensure it works as expected on the new v3 API update while also working on the older api version. Please see the corresponding Height ticket for the schemas used to validate. Additionally, I was able to gain confidence in these changes due to the customer in the linked Issue claiming the fix resolved upgrade issues they were experiencing.

Additionally, I was able to validate that the impacts downstream models updated within this PR saw consistent results when using the old and new versions of the package. See the below screenshots for a few validations. FYI the old schema is listed as zz_dbt_joe_hubspot_old_api and the new is simply zz_dbt_joe_hubspot.

• Old hubspot__engagements model vs New

  • 
  • 

• I also was able to validate that for all the engagement__[type] end models that the row counts were inconsistent between versions which was expected.

Standard Updates

Please acknowledge that your PR contains the following standard updates:

• Package versioning has been appropriately indexed in the following locations:
  • [X] indexed within dbt_project.yml
  • [X] indexed within integration_tests/dbt_project.yml
• [X] CHANGELOG has individual entries for each respective change in this PR
• [X] README updates have been applied (if applicable)
• [n/a] DECISIONLOG updates have been updated (if applicable)
• [X] Appropriate yml documentation has been added (if applicable)

dbt Docs

Please acknowledge that after the above were all completed the below were applied to your branch:

• [X] docs were regenerated (unless this PR does not include any code or yml updates)

If you had to summarize this PR in an emoji, which would it be?

🙃