[Bug] missing source column causing model error

[kcraig-ats]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

We're running into an error running stg_hubspot__deal. The property_closedate column is the issue because it is not included in our instance of the deal source table. We have hubspot__deal_pass_through_columns enabled, which bypasses the macro CTE where those columns are staged.

Relevant error log or model output

No response

Expected behavior

The staged columns created in the macro CTE would be used irrespective of the use of the hubspot__deal_pass_through_columns.

dbt Project configurations

hubspot__pass_through_all_columns: true

Package versions

package: fivetran/hubspot
version: [">=0.8.0", "<0.9.0"]

What database are you using dbt with?

redshift

dbt Version

1.3

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @kcraig-ats thanks for opening this issue. This is an interesting scenario 🤔.

I agree that it would be best if the get_deal_macro would be able to be used irrespective of the hubspot__pass_through_all_columns variable being set to true. However, the get_deal_columns only includes a fraction of the fields. Therefore, if we used the macro, we would not be able to select all fields from your source table. In order to leverage the macro, we would need to include all the possible fields. Which is unfortunately not realistic due to the customizable nature of these HubSpot sources.

I am curious if you have any insight into why you do not have the property_closedate field in your source deal table? I was under the impression this was standard field that would represent the closed date for deals. Do you have a different field that represents this same value?

Regardless, you pose a valid bug and one we should address in the next release of the package. In the meantime, I will brainstorm with the team so viable avenues to ensure the package works when using the all column variable, while still ensuring the required fields are populated regardless of their presence in the source.

[kcraig-ats]

Hi @fivetran-joemarkiewicz - Thanks for responding so quickly! Yeah, my guess is this an edge case. The field is standard in HubSpot but only contains null data at the moment. My understanding is that the connector skips those fields.

[fivetran-jamie]

the fix for this should officially be in v0.9.0 of the package!