snowplow / snowbridge

For replicating streams across clouds, accounts and regions
Other
15 stars 7 forks source link

Rationalise transformations and transformation config #169

Closed TiganeteaRobert closed 2 years ago

colmsnowplow commented 2 years ago

Documenting a key decision here after some discussion.

The initial implementation of scripting transformations was restricted to only one JS or Lua 'engine' per configuration.

The decision was then made to allow users to define multiple functions in the script, so that they may have multiple scripting transformations given that constraint. They provide the function name in configuration, the single engine has all functions, and then each instance of a transformation spins up a vm using the provided function name.

With the changes to configuration in this issue, we found a way to implement multiple engines. The decision was made to change the above pattern so that each script has a single engine, and the script must have a main() function, which is what gets executed. Configuration for both script and engine are kept together, and it should be simpler to configure.

There may be requirements to change this pattern in future, but this decision seems to simplify the implementation and config somewhat.