snowplow / snowbridge

For replicating streams across clouds, accounts and regions
Other
15 stars 7 forks source link

Feat/transform #42

Closed colmsnowplow closed 3 years ago

colmsnowplow commented 3 years ago

Adds a transformation package and integration into the main functions.

The design is intended to be generalisable to applying any kind of transformation upon any kind of data (including filters), but this PR only provides an implementation for enriched Snowplow events' transformation to JSON.

Three things I'll point out as being uncertain of (mostly documented inline also):

  1. We edit the input when we perform a transformation (I believe because messages takes a pointer?)
  1. The unit tests are tricky because of point 1. I'm unsure if some of the equivalence checks are actually a tautological pass because the input is edited by the transformation function.

  2. The specific implementations of transformations don't currently return any error. The assumption here is that any error we hit would result in a message going to the Invalid slice in the result (which contains an error message).

colmsnowplow commented 3 years ago

Ok @jbeemster I think I've addressed everything now. Also have added metrics. :)

colmsnowplow commented 3 years ago

Closing as changes now pulled into #49