search

practo / tipoca-stream

Near real time cloud native data pipeline in AWS (CDC+Sink). Hosts code for RedshiftSink. RDS to RedshiftSink Pipeline with masking and reloading support.
https://towardsdatascience.com/open-sourcing-tipoca-stream-f261cdcc3a13
Apache License 2.0
47 stars 5 forks source link

Retrying release for an already released table #246

Open alok87 opened 3 years ago

alok87 commented 3 years ago

Operator released a table, then the async status update did not happen somehow, and since status did not get updated in the next reconcile it found the topic is not released and tried to release it again and got stuck in this loop ever since.

Need to make sure the status update happen, then only consider the release as valid!

alok87 commented 3 years ago

Guessing: The issue is happening because the release query to rename the table was stuck due to some reason, and the reconciler process got stuck waiting for it, when the reconciliation timed out, it went to try again, but the query has done its job in Redshift and the status could not get updated for it as the reconciler timedout!

alok87 commented 3 years ago

Found: status update was not happening after multiple release tries for a table, this could be related to status API not giving 200 for a particular payload or something. Need to tie the release success with status getting updated success!

Reducing the default of MaxTopicRelease from 50 to 5 and marking the issue as P3.