search

practo / tipoca-stream

Near real time cloud native data pipeline in AWS (CDC+Sink). Hosts code for RedshiftSink. RDS to RedshiftSink Pipeline with masking and reloading support.
https://towardsdatascience.com/open-sourcing-tipoca-stream-f261cdcc3a13
Apache License 2.0
47 stars 5 forks source link

Mutilple table release one per reconcile is slow #141

Closed alok87 closed 3 years ago

alok87 commented 3 years ago

Possible cause 1: Mutilple table release one per reconcile is slow since it sinks all the sink groups again based on the state change. Treat multiple topic release as one activity and save time

Possible cause2: fetch to kafka is not happening due to https://github.com/practo/tipoca-stream/issues

alok87 commented 3 years ago

This is related to #140 made it faster in https://github.com/practo/tipoca-stream/pull/125/commits/dc56ca03f3296078eab3fd0889ee3faed308c1a0