adiom-data / dsync

Database synchronization tool
https://adiom.io
GNU Affero General Public License v3.0
5 stars 1 forks source link

Progress reporting switches to change stream before all initial data copy writes are done #61

Closed alex-thc closed 1 month ago

alex-thc commented 2 months ago

Since we rely on the reader to display progress (CLI/Web), it's possible that the reader has already finished reading data for initial data copy, but the writer still has its queues full in ParallelWriter. Thus the UI will switch to ChangeStream mode prematurely. This can be significant in case of an extremely slow batch writer, such as Cosmos DB