search

awslabs / dynamodb-streams-kinesis-adapter

The Amazon DynamoDB Streams Adapter implements the Amazon Kinesis interface so that your application can use KCL to consume and process data from a DynamoDB stream.
Apache License 2.0
99 stars 37 forks source link

Is there a version of the adapter for KCL v2.x #42

Open artur-jablonski opened 3 years ago

artur-jablonski commented 3 years ago

As per title. I am looking at consuming dynamodb stream via KCL and ended up here. I need to use KCL v2.x, but this library seems to be dealing with KCL v1.x. Is there an equivalent implementation for KCL v2.x?

gjesse commented 3 years ago

I'm not a member of the project but i went through this a while back and concluded that, no, there is not.

artur-jablonski commented 3 years ago

Same conclusion. On the good side, seems like dynamodb has a new feature to stream directly to Kinesis.

gjesse commented 3 years ago

whoa.. thanks for the heads up. that looks great!

mustafaakin commented 3 years ago

Why pay extra for Kinesis when DynamoDB Streams exist? I looked at the how to implement my own but looks complex for now.

dacevedo12 commented 2 years ago

Any updates?

https://github.com/awslabs/amazon-kinesis-client/issues/451

olbat commented 1 year ago

@artur-jablonski , I think that Kinesis can only be an easy alternative for some use cases.

Here are some key differences:

Change data capture for DynamoDB Streams:

DynamoDB Streams helps ensure the following:

  • Each stream record appears exactly once in the stream.
  • For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Kinesis Data Streams to capture changes to DynamoDB:

The Kinesis data stream records might appear in a different sequence than the item changes occurred. The same item notifications might also appear more than once in the stream. You can check the ApproximateCreationDateTime attribute to identify the approximate order that the item modifications occurred, and to identify duplicate records. ApproximateCreationDateTime indicates the time of the modification in milliseconds.

Therefore, depending the the use-case, one might not easily be able to trade one for the other (e.g. if the application needs the guarantee that messages appear in order of modification in the change stream).