[Feature] Include Description for HubSpot Custom Properties in dbt Models

[Mick1406]

Is there an existing feature request for this?

• [X] I have searched the existing issues

Describe the Feature

📍 Current state

Although it is possible to include pass through columns in the configuration to retrieve hubspot custom properties, the package currently does not pass through the description of custom properties defined in HubSpot into the dbt models. This missing information can be crucial for better understanding the context and usage of custom properties, especially when analyzing the data or sharing documentation.

⚙️ Feature Request

We would love, and be eternally grateful 😄, if the dbt HubSpot package could include the description of custom properties from HubSpot in the generated dbt models.

🧬 Expected Benefits

• Improved Data Understanding: Having property descriptions readily available in the dbt models will provide users with better insights into the meaning and purpose of each custom property, facilitating more effective data analysis.
• Enhanced Data Documentation: The property descriptions will be available through the dbt docs site, enabling users to access a comprehensive data dictionary for both the source and modeled HubSpot data.

Describe alternatives you've considered

• Generate the corresponding yml file for the hubspot table (e.g. hubspot__deals) and input the descriptions for each pass through columns.

🧪 Other implementation considerations:

• The property descriptions can be retrieved through the HubSpot API when fetching metadata for custom properties.
• The descriptions can be included in the dbt models as part of the corresponding field's metadata or as an additional column in the data model.

Are you interested in contributing this feature?

• [ ] Yes.
• [x] Yes, but I will need assistance and will schedule time during your office hours for guidance.
• [ ] No.

Anything else?

No response

[fivetran-joemarkiewicz]

Hi @Mick1406 thanks for opening this feature request!

I definitely see the value of passing through definitions for the passthrough columns. However, I see this as a bit of a challenge as the dbt yml files are not very friendly to dynamic use cases (I have been a huge advocate for this, but it seems it is not entirely possible). Therefore, I don't even know if it would work where the field definitions were included in a variable of sorts. I also don't believe we would be able to retrieve data from an API within the dbt package (albeit I have never tried and that would be wicked, but I don't know how this would be accomplished).

What has usually been the route we suggest when customers want to modify the definitions we provide in the package is to perform a source override. However, this does not allow us to add new fields and definitions (which is what you are wanting to do). It may be that the best approach to make is to possibly open a FR on dbt-core to support more dynamic yml or for more flexibility with source overrides.

I am definitely open to discussing this more. However, each time I go down this route I usually end up hitting a towering wall that stops me in my tracks. Let me know if you have any other thoughts!

[Mick1406]

Hi @fivetran-joemarkiewicz 👋,

Thank you for your swift response and for considering my feature request!

I completely understand the complexities involved in making the dbt yml files dynamic and the limitations with regard to API integration within the dbt package (I too have no idea how this would be accomplished). The conclusion is that given the current constraints, the approach I suggested might not be feasible within the existing framework.

Your suggestion of exploring enhancements, or increased flexibility, within dbt-core for handling more dynamic yml files or source overrides might be the best approach.

I think it makes sense to close this FR for now. I will check with dbt-core whether or not they are currently working on handling dynamic yml files (or at least whether it's on their roadmap)...

Thanks!