Make it possible for processor to return multiple records

ConduitIO / conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

https://conduit.io

Apache License 2.0

376 stars 44 forks source link

Make it possible for processor to return multiple records #636

Open hariso opened 1 year ago

hariso commented 1 year ago

Feature description

Currently, a processor accepts a single record as an input, and returns a single record as an output.

It would be good if the processor would return multiple records for the following reasons:

According to @ahmeroxa it's fairly when going to relational databases (e.g. from a denormalized to a normalized data model)
Some other tools use that too (e.g. Benthos), so doing a switch to Conduit would be easier.

simonl2002 commented 1 year ago

@ahmeroxa is there a actual need for this currently? Otherwise going to delay this until after 0.8.0 when we will tackle splitting and combining records more wholistically.

ahmeroxa commented 1 year ago

@simonl2002 it's not currently blocking any active projects, so it can wait.