ConduitIO / conduit-connector-sdk

SDK for Conduit connectors written in Go
Apache License 2.0
11 stars 4 forks source link

End-to-end batching #193

Closed lovromazgon closed 4 weeks ago

lovromazgon commented 1 month ago

Description

This PR adds the source middleware SourceWithBatch which enables source connectors to send records in batches to Conduit, which improves the performance.

The source connector developer doesn't have to change anything, existing connectors can benefit from this middleware. Although, notice that the Source middleware now also allows the implementation of ReadBatch, which allows the connector developer to return whole batches directly.

This PR also updates the destination connector to accept and correctly handle batches received from Conduit. Previously the batch was broken up into separate records. Now a batch is handled as is.

[!NOTE] These changes should not have an impact on the current behavior of connectors. However, they allow us to significantly improve the performance once we change the Conduit internals to handle batches end-to-end.

Quick checks: