search

koralium / flowtide

Streaming integration engine
https://koralium.github.io/flowtide/
Apache License 2.0
28 stars 2 forks source link

Change from ReadWriteFactory to ConnectorManager #460

Closed Ulimo closed 4 months ago

Ulimo commented 4 months ago

This is a required task fask for #456

This change will require connectors to be added using IConnectorSinkFactory, IConnectorSourceFactory and IConnectorTableProviderFactory.

These change are done for two reasons:

  1. Allow seperation of plan modification by connectors, a connector may want to read in parallel with multiple read sources. This is not possible today.
  2. Allow a source connector to provide a table provider that can be used when creating a plan from sql. This simplifies startup configuration where a source is added once for both creating the plan and also executing the plan.