snowplow / snowbridge

For replicating streams across clouds, accounts and regions
Other
15 stars 7 forks source link

Move each component's configuration into its pkg source code #95

Open colmsnowplow opened 2 years ago

colmsnowplow commented 2 years ago

In https://github.com/snowplow-devops/stream-replicator/pull/93 we propose a new structure for sources, whereby each source is its own package, which provides a configuration object for itself, and a generic sourceconfig package provides the GetSource function - which takes as an input a list of supported sources, and returns the source based on the provided configuration.

This structure allows us to exclude kinesis source from a GCP build and avoid the kinsumer licence issue, but it also lends itself to a more pluggable project structure overall.

The proposal here is to implement this structure for transformations and targets, moving each transformation/target's configuration definition to live alongside its own source code.

Then, if we can modify config parsing to be less rigid, and allow it to take arbitrary configurations, we can allow for a fully customisable 'pluggable' structure. If this is successful, it would be possible to integrate fully custom sources, transformations and targets, by writing a package, importing it, and building a main function yourself, which looks something like this:

func main() {

    sourceConfigPairs := []sourceconfig.SourceConfigPair{myCustomSource.customConfigDefinition}
        transformConfigPairs := ...
        targetConfigPairs := ...

    cli.RunCli(sourceConfigPairs, transformConfigPairs, targetConfigPairs)
}