search

snowplow / snowbridge

For replicating streams across clouds, accounts and regions
Other
15 stars 7 forks source link

Split build to exclude kinsumer for GCP deployments (closes #71) #93

Closed colmsnowplow closed 2 years ago

colmsnowplow commented 2 years ago

@jbeemster the changes you suggested on our call have now been made. Let me know what you think. :)

colmsnowplow commented 2 years ago

Note - most recent commit adds missing copyright notice with update for 2022. To make a review less painful I've kept it in a different branch for now but the release will also update all the other copyright notices.

colmsnowplow commented 2 years ago

@jbeemster @paulboocock I have refactored to address the points you both raised. I think this is much better.

Long story short, we now provide the GetSource() function in pkg/source, which takes a config and a list of supported targets for a given build.

Each source defines one of these:

type SourceConfigPair struct {
    SourceName       string
    SourceConfigFunc SourceConfigFunction
}

GetSource iterates the provided sources, and if config.Source matches SourceName, it returns the result of SourceConfigFunc.

I think this makes each build's code more transparent, and also if we figure out how to make config parsing more flexible, this structure could allow for custom builds with custom built sources. (And using the same pattern, transformations and targets too).

Apologies for the amount of change since last iteration, I'm hoping it'll be clear what I've done here.