search

trustpilot / kafka-connect-dynamodb

A Kafka Connect Source Connector for DynamoDB
MIT License
56 stars 32 forks source link

Support for bulk mode along with existing incrementing mode #6

Closed NiramayVaidya closed 4 years ago

NiramayVaidya commented 4 years ago

Incrementing mode for the JDBC source connector only publishes a newly inserted row in a table to the topic based upon an auto-incrementing column. This source connector goes a step further to also publish a modified row in a table as another record to the topic. Both these actions can be recorded with the use of DynamoDB Streams since it maintains any kind of changes to the data in the table including inserts and updates for a fixed period of time. Is there any way to add support for the bulk mode using which every time the connector is started, it publishes all the rows present in a table to the topic?

raimondast commented 4 years ago

If I understood you correctly you are referring to "back-filling" support. If so, then yes, backfilling/initial-sync is supported as noted here: https://github.com/trustpilot/kafka-connect-dynamodb#notable-features