search

sanjuthomas / kafka-connect-gcp-bigtable

Kafka Sink Connect to GCP Bigtable - https://www.confluent.io/hub/sanjuthomas/kafka-connect-gcp-bigtable
http://sanjuthomas.com
MIT License
8 stars 8 forks source link

Tombstone support for delete #12

Open marcospassos opened 4 years ago

marcospassos commented 4 years ago

Any plans to add support for deleting records through tombstone?

sanjuthomas commented 4 years ago

@marcospassos , Could you please explain your requirement?

marcospassos commented 4 years ago

Sure. I'd like to sink deleted records whenever a tombstone record is received. See how the JDBC connector handles tombstone records.

sanjuthomas commented 4 years ago

@marcospassos Sure! I will add that over this weekend. Thank you!

sanjuthomas commented 4 years ago

@marcospassos, Would you able to help me with the testing? I ran out of google cloud credit.

The requested feature is available in the following branch -

https://github.com/sanjuthomas/kafka-connect-gcp-bigtable/tree/feature/tombstone-record-delete

I'm using the BulkMutation so I'm not sure how that would behave if we have delete and add for the same row key in the same batch. That would be an additional test case that we have to execute in the integration environment.

Please let me know.