search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.87k stars 3.82k forks source link

[source-dynamodb] : Use Dynamodb Stream for incremental load #36044

Open sivankumar86 opened 4 months ago

sivankumar86 commented 4 months ago

Connector Name

source-dynamodb

Connector Version

0.2.3

What step the error happened?

During the sync

Relevant information

It scan entire table for incremental load. Could you implement stream to read incremental load ?

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

Relevant log output

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

Contribute

marcosmarxm commented 4 months ago

This source won't be prioritize for now @sivankumar86 so it can take some time until get this solved. If you want to contribute please feel free to ping me in Slack.