search

Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
748 stars 70 forks source link

Progress now account for more things #228

Closed shuhaowu closed 3 years ago

shuhaowu commented 3 years ago

Related: #226

Added the current number of active data iterators as well as the number of rows copied per table into the progress struct. This will allow us to track the performance of a Ghostferry move with a bit more precision.

The code is a bit crappy right now as we get this functionality working and measuring in real moves. The reason it's crappy is because the StateTracker now has two roles: tracking the state which is vital for interrupt and resume, as well as measuring some optional performance metrics. These two objectives ideally would be refactored so they're not as closely intertwined.

xliang6 commented 3 years ago

I agree with Forrest that we can hold off the code for EstimatedBytes till we know and have time to implement it. This PR looks good otherwise.