search

koralium / flowtide

Streaming integration engine
https://koralium.github.io/flowtide/
Apache License 2.0
28 stars 2 forks source link

Change startup configuration of Flowtide with Dependency Injection #456

Closed Ulimo closed 4 months ago

Ulimo commented 4 months ago

Today it is a bit complicated to setup a stream when getting started.

A new project should be created called "FlowtideDotNet.DependencyInjection" The "AspNetCore" project will reference this project and use it for the startup. The new project is created to not require a dependency on AspNetCore.

The suggestion on how to setup Flowtide looks like the following:

services.addFlowtideStream()
 .AddSqlFileAsPlan("query.sql")
 .AddConnectors((connectorManager, serviceProvider) => {
     connectorManager.AddSqlServerSource("connStr");
     connectorManager.AddElasticsearch(options);
  })
 .AddStorage(b => {
     b.UseTemporaryDevelopmentStorage();
     b.UseAzureStorage(connectionString, "containerName");
  })

How connectors are handled needs to be changed, today they are added to the ReadWriteFactory as functions that create the source or sink.

Instead each connector should create a factory class that can implement IConnectorSourceFactory, IConnectorSinkFactory, IConnectorSqlTableProvider.

When adding a source, the factory added should atleast implement IConnectorSourceFactory. If it also implements IConnectorSqlTableProviderFactory, it will be used when created the substrait plan.