snowplow-incubator / snowplow-snowflake-loader

Loads Snowplow enriched events from S3 into Snowflake
11 stars 9 forks source link

Add extraction of joinkeys from nested structures #53

Open colmsnowplow opened 6 years ago

colmsnowplow commented 6 years ago

If a joinkey is in a nested field (eg. the page_view id in an array), modeling the data around performance can be a challenge.

I suggest we add some way of indicating a joinkey in a custom event, and extract those values into their own column alongside loading the data as a nested object.

(A 'joinkey' option in the schema feels most straightforward).

Have spoken about it before, adding as an issue for future reference.

alexanderdean commented 6 years ago

Really neat idea!