search

snowplow / enrich

Snowplow Enrichment jobs and library
https://snowplowanalytics.com
Other
21 stars 39 forks source link

Stream: consider changing the enriched event format #182

Open chuwy opened 4 years ago

chuwy commented 4 years ago

The output of Scala Kinesis Enrich is super-brittle because it's a TSV. Every time we upgrade the TSV, we have to also upgrade any downstream apps.

Why not make it so that Scala Kinesis Enrich emits a self-describing JSON which versions the enriched event? That way, we could upgrade the downstream apps so that they support both the old and the new versions of the enriched event.

This would be super-flexible and would allow us to innovate on the format of the enriched event much faster.

It should also be easier for ops, because a single stream of enriched events can contain multiple different versions of the enriched event schema...

chuwy commented 4 years ago

Migrated from https://github.com/snowplow/snowplow/issues/1470 (comments are auto-generated)