The common-streams Kinesis source suffers from a problem where we don't quite achieve at-least-once processing semantics near the end of a shard. The problem was in the 3rd-party fs-kinesis library, and it is not easy to fix with any small code change to that library.
Sorry I cannot provide a link here to back that up -- it is documented internally at Snowplow.
This PR re-implements our Kinesis source from scratch, this time without a dependency on fs2-kinesis. The biggest difference is the way we block the shardEnded method of the KCL record processor, until all records from the shard have been written to the destination.
The common-streams Kinesis source suffers from a problem where we don't quite achieve at-least-once processing semantics near the end of a shard. The problem was in the 3rd-party fs-kinesis library, and it is not easy to fix with any small code change to that library.
Sorry I cannot provide a link here to back that up -- it is documented internally at Snowplow.
This PR re-implements our Kinesis source from scratch, this time without a dependency on fs2-kinesis. The biggest difference is the way we block the
shardEndedmethod of the KCL record processor, until all records from the shard have been written to the destination.