search

awslabs / dynamodb-streams-kinesis-adapter

The Amazon DynamoDB Streams Adapter implements the Amazon Kinesis interface so that your application can use KCL to consume and process data from a DynamoDB stream.
Apache License 2.0
97 stars 37 forks source link

Migration to sdk2 #22

Open posac opened 5 years ago

posac commented 5 years ago

Hi,

Do you have plan to migrate to aws sdk2 ? :)

Michal

aggarwal commented 4 years ago

We currently don't have any plans to migrate to AWS SDK 2.0 in the next 6 months. SDK 2.0 can run alongside SDK 1.x in a JVM. What is your use case to migrate DynamoDB Streams adapter to 2.0?

camerondavison commented 4 years ago

I would say that the KCL v2.0 has a nicer interface, and if you are using it for other kinesis activity it would be nice to not have to learn both apis.

adrian-baker commented 4 years ago

KCL 1.x is quite buggy, maybe 2.x is less so?

aggarwal commented 4 years ago

KCL 1.x is quite buggy, maybe 2.x is less so?

DynamoDB continues to work on improvements to KCL 1.x and DDB Streams Kinesis adapter, including bug fixes. Kinesis also supports both 1.x and 2.x and continues to work on bug fixes for both versions.

anoopgarlapati commented 4 years ago

As noted in https://github.com/awslabs/dynamodb-streams-kinesis-adapter/issues/22#issuecomment-536130237, I have tried migrating KCL to 2.x and during my migration I have encountered that there is no equivalent for com.amazonaws.services.dynamodbv2.streamsadapter.model.RecordAdapter which can transform a KinesisClientRecord object from KCL 2.x. I need this to get the old and new images from the stream. This was reported by another user in kinesis-client repo https://github.com/awslabs/amazon-kinesis-client/issues/714 and was directed to ask here. It seems like there is a gap in using KCL 2.x with 1.x of dynamodb-streams-kinesis-adapter so is there a workaround for this?

aggarwal commented 4 years ago

DynamoDB Streams does not support KCL 2.x today, and building this support is a larger effort than implementing 2-3 classes. There is no recommended workaround for this at this time, and we recommend using KCL 1.13.3 with version 1.5.1 of the adapter as of now. We continue to support customers who are using this combination.

etspaceman commented 4 years ago

What is the timeline for updating this library to use V2?

kyeotic commented 1 year ago

Is there any update on this? It seems like this is supposed to be the official path, but support does not exist.

The docs for the V2 DynamoDB Streams say

Amazon DynamoDB Streams provides API actions for accessing streams and processing stream records. To learn more about application development with Streams, see Capturing Table Activity with DynamoDB Streams in the Amazon DynamoDB Developer Guide.

If you follow that link you end up here, where it says

Using the Amazon Kinesis Adapter is the recommended way to consume streams from Amazon DynamoDB.

So the V2 SDK docs lead you back here, where support for v2 is not supported. While a timeline for v2 support would be nice, it seems like the docs should at least warn potential upgraders of the missing support.

artembilan commented 1 year ago

Also stuck on this. I wonder if after this feature on DynamoDB we don't this adapter any more...

olbat commented 1 year ago

@artembilan I think that the feature you mentioned is not always a good alternative to native DynamoDB Streams

The integration of DynamoDB with Kinesis data streams to implement change data capture is absolutely not equivalent to the native implementation of DynamoDB Streams. These are two key differences:

Change data capture for DynamoDB Streams:

DynamoDB Streams helps ensure the following:

  • Each stream record appears exactly once in the stream.
  • For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Kinesis Data Streams to capture changes to DynamoDB:

The Kinesis data stream records might appear in a different sequence than the item changes occurred. The same item notifications might also appear more than once in the stream. You can check the ApproximateCreationDateTime attribute to identify the approximate order that the item modifications occurred, and to identify duplicate records. ApproximateCreationDateTime indicates the time of the modification in milliseconds.

Therefore, depending the the use-case, one might not easily be able to trade one for the other (e.g. if the application needs the guarantee that messages appear in order of modification in the change stream).

So imo, support of native DynamoDB Streams in KCL v2 & the stream adapters looks like a rather important feature.

@aggarwal, do we have any update about a timeline for the support of this feature? Thanks in advance 👍

JHTosas commented 6 months ago

AWS SDK v1 will enter maintenance mode in July 31st, 2024, and reach end-of-life on Dec 31st, 2025. I really hope the migration is done before that. https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/