PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.17k stars 88 forks source link

Validate mirror: account for resync more #2044

Closed Amogh-Bharadwaj closed 3 weeks ago

Amogh-Bharadwaj commented 3 weeks ago

Resync currently does:

  1. Drop mirror
  2. Create mirror - which in turn does Validate mirror

If validate mirror fails here for some reason (an example of which is fixed in this PR), then the UX is completely messed up because the mirror is dropped and the user does not have any idea of what went wrong

This PR calls validate mirror explicitly in the Resync endpoint before dropping the mirror, returning any validation errors. The subsequent validate call in Create Mirror is put behind a !resync guard so that this new mirror gets created

Second thing this PR does is not perform destination table emptiness and engine check for Clickhouse mirror validation in case of a resync, because those tables will get swapped out with the _resync tables whose structure and emptiness PeerDB fully controls because of CREATE OR REPLACE

Thirdly, this PR improves UI/UX of resync in PeerDB UI to now keep the resync dialog open so that all user-facing messages are visible. Resync and close buttons are disabled until resync has been kicked off. Messages changed to be more accurate (Ex: Resyncing... changed to Preparing resync..)

Screenshot 2024-09-05 at 4 10 29 AM Screenshot 2024-09-05 at 4 33 49 AM