search

Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
693 stars 65 forks source link

Dynamic reconfiguration for DataIterationBatchSize #316

Closed yevsafronov closed 2 years ago

yevsafronov commented 2 years ago

Re: https://github.com/Shopify/ghostferry/issues/265

As part of the above issue I was looking into a way to implement dynamic configuration via the control server. This PR adds a new endpoint to the ControlServer which allows dynamic reconfiguration, currently it only supports updating the DataIterationBatchSize value.

Another important change is the move of ControlServer into Ferry as an optional component, disabled by default and enabled for copydb. This will allow to use the dynamic reconfiguration and other ControlServer endpoints outside of copydb.

yevsafronov commented 2 years ago
  1. We need to display the current batch size for each table in the control server ui. The reason we display for each table is because in the future we might automatically change them.

should this be a new issue?

Lastly, this architecture is fine for now. That said, there are some possibility that we might automatically adjust the batch size of Ghostferry via a simple algorithm. I'm not quite sure how the pointer to batch size would interact with that. We should probably at least think about this a bit and decide to ignore it until we actually implement that feature (maybe leave a comment in this PR about it).

we might want to add a parameter to control the way we handle batch size, i.e MANUAL vs AUTOMATIC

shuhaowu commented 2 years ago

I think we should add the display in this issue. Should only be a few lines of code? Also since we're making it updatable, we should display it somewhere so we know the update is successful at least.

yevsafronov commented 2 years ago

I think we should add the display in this issue. Should only be a few lines of code? Also since we're making it updatable, we should display it somewhere so we know the update is successful at least.

added batch size to to the ui