A fully incremental model, that transforms raw web & mobile event data generated by the Snowplow JavaScript & mobile trackers into a series of derived tables of varying levels of aggregation.
This PR in theory adds the ability to run snowflake on a lake loader external (iceberg) table. Currently it does not work because you can't write structured columns to a regular table, and dbt-snowflake doesn't yet support writing to iceberg. In theory it would work, but short of casting every single field within a context and unstruct to a varchar and back again, I am currently not able to test it.
I've worked around this similar to how we do in web with redshift not having a "* except" feature, by getting all columns from the CTE and then casting them all individually. This feels non-ideal, but at least in my basic testing does seem to allow the models to run at least.
I would suggest:
Further testing to ensure everything is working as expected (test using
snowplow__database: snowplow_dev1
snowplow__events_table: "EVENTS_ICEBERG" # Only set if not using 'events' table for Snowplow events data
snowplow__enable_web: true
snowplow__enable_mobile: false
snowplow__backfill_limit_days: 1
snowplow__start_date: '2024-06-13'
snowplow__enable_yauaa: true
snowplow__enable_ua: true
If you do release this, announce limited support for it - for example thing such as passthrough fields and custom SQL in events this run may not work as expected.
Description
This PR in theory adds the ability to run snowflake on a lake loader external (iceberg) table. Currently it does not work because you can't write structured columns to a regular table, and dbt-snowflake doesn't yet support writing to iceberg. In theory it would work, but short of casting every single field within a context and unstruct to a varchar and back again, I am currently not able to test it.
I've worked around this similar to how we do in web with redshift not having a "* except" feature, by getting all columns from the CTE and then casting them all individually. This feels non-ideal, but at least in my basic testing does seem to allow the models to run at least.
I would suggest:
What type of PR is this? (check all applicable)
Related Tickets & Documents
https://snplow.atlassian.net/browse/PE-6472
Checklist
Added tests?
Added to documentation?
[optional] Are there any post-deployment tasks we need to perform?
[optional] What gif best describes this PR or how it makes you feel?