search

fivetran / dbt_fivetran_log

Data models for Fivetran's internal log connector built using dbt.
https://fivetran.github.io/dbt_fivetran_log/
Apache License 2.0
30 stars 24 forks source link

[Bug] JSON parsing error in Redshift #113

Closed fivetran-reneeli closed 6 months ago

fivetran-reneeli commented 7 months ago

Is there an existing issue for this?

Describe the issue

Error downstream in fivetran_platform_connector_status and fivetran_platform_audit_table due to the message_data the log table getting truncated due to lengthy json value. The incomplete json is causing the model runs to fail.

In order to support these json objects in Redshift the SUPER type is necessary but it is complicated.

As a workaround, we should update in the package to filter message_data first for only necessary keys prior to any parsing (event_subtype='status')

Relevant error log or model output

No response

Expected behavior

json to be parsed correctly and package to run

dbt Project configurations

na

Package versions

na

What database are you using dbt with?

redshift

dbt Version

na

Additional Context

No response

Are you willing to open a PR to help address this issue?

fivetran-catfritz commented 6 months ago

This has been addressed in release v1.5.0, which is now live on the dbt hub. Closing the issue!