search

fivetran / dbt_hubspot_source

Data models for Hubspot built using dbt.
https://fivetran.github.io/dbt_hubspot_source/
Apache License 2.0
31 stars 30 forks source link

[Feature] Include Deleted Deal Pipelines & Deal Pipeline Stages #93

Closed jamesrayoub closed 1 year ago

jamesrayoub commented 1 year ago

Is there an existing feature request for this?

Describe the Feature

HubSpot hard-deletes deal pipelines and deal pipeline stages from the workspace but Fivetran does capture using _fivetran_deleted. For historical deal stage activity, in instances where the deal stage may have been deleted, it would be helpful to have the added context as to what that previous stage was rather than simply returning a null value.

A simple fix would be to remove where not coalesce(_fivetran_deleted, false) from both stg_hubspot__deal_pipeline and stg_hubspot__deal_pipeline_stage. Given _fivetran_deleted is provided in the resulting view for each, the end user can filter out deleted deal stages when applicable

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

Anything else?

No response

fivetran-joemarkiewicz commented 1 year ago

Hey @jamesrayoub thanks for opening this issue and I think your proposal makes a lot of sense!

My one concern is if this will impact that tests associated with these staging models and I wonder what the downstream impacts may be 🤔. However, a better approach may be to exclude the filters here so all may be able to take advantage of the history, and then filter out in the downstream models within the dbt_hubspot package.

I noticed you are open to contributing to the package! 🎉 I feel it would be appropriate to apply this update as long as we can ensure the tests are still relevant and any downstream models within dbt_hubspot properly filter out the historical records.

If you are not interest in opening the PRs, we can fold this into a future sprint when we come back in the new year!

jamesrayoub commented 1 year ago

Yes, happy to contribute here. Seems like dbt_hubspot is using the is_active flag on both the deal_pipeline and deal_pipeline_stage objects within the int_hubspot__deals_enhanced and hubspot__deal_stages models. My thought is to repurpose that field by evaluating whether or not the pipeline or stage is deleted; in our instance, is_active is always true so makes sense since it's hard deleted and never updated. I'll put together a PR!

fivetran-joemarkiewicz commented 1 year ago

Closing this issue as PR #96 addressed this issue. Thanks again for raising this to our team and helping work through a resolution.