This PR adds the source middleware SourceWithBatch which enables source connectors to send records in batches to Conduit, which improves the performance.
The source connector developer doesn't have to change anything, existing connectors can benefit from this middleware. Although, notice that the Source middleware now also allows the implementation of ReadBatch, which allows the connector developer to return whole batches directly.
This PR also updates the destination connector to accept and correctly handle batches received from Conduit. Previously the batch was broken up into separate records. Now a batch is handled as is.
[!NOTE]
These changes should not have an impact on the current behavior of connectors. However, they allow us to significantly improve the performance once we change the Conduit internals to handle batches end-to-end.
Description
This PR adds the source middleware
SourceWithBatch
which enables source connectors to send records in batches to Conduit, which improves the performance.The source connector developer doesn't have to change anything, existing connectors can benefit from this middleware. Although, notice that the
Source
middleware now also allows the implementation ofReadBatch
, which allows the connector developer to return whole batches directly.This PR also updates the destination connector to accept and correctly handle batches received from Conduit. Previously the batch was broken up into separate records. Now a batch is handled as is.
Quick checks: