Improve storage engine migration

[etschannen]

Changing storage engines on a live clusters currently has a number of different problems. The way it is currently implemented, data distribution will mark all storage servers as unhealthy, and then recruit new storage servers with the new storage engine on all of the storage processes. It will then issue a ton of data movement to move the data from the old storage servers to the new storage servers.

The fact that the processes are each running two storage servers causes performance problems. It also can cause disk space issues, because the storage servers take a very long time to shrink their file size.

Proposed design:

• A process cannot become a storage server if it is already running as a storage server
• Data distribution does not consider all processes running the old storage engine as unhealthy
• Data distribution always has exactly one process with the old storage engine that it considers unhealthy

This design should completely move the data off of one process because changing its storage engine. This will then repeat until every process has changed over to the new engine.

[xumengpanda]

Add some clarification on this issue:

The problem: When fdbcli uses configure [storage_engine] , the current system (data distribution) marks all storage servers (which uses the old storage engine) as unhealthy and immediately recruits new storage engines on the storage server process. This causes a storage server process has two storage roles: one for the old storage engine, and the other for the new storage engine. But the storage server's memory limit is still the memory limit for one storage engine. Running two storage engine roles on a process configured for one storage engine is a bad idea.

The proposed solution: 1) Still allow fdbcli to configure the new storage engine as it does now; 2) Mark the old storage engine role unhealthy one by one instead of all at once; This avoid creating too much immediate data movement and impacting the online system; 3) Only when a storage process has no storage engine role, can the storage process be recruited for a new storage role. This avoids the above problem that a storage process runs with two storage engine roles. This requires changes on how to recruit the storage engine.

[xumengpanda]

To completely solve this issue, we need:

1. Ensure at most one storage server per process, which requires: i) change simulation test such that it will not change storage engine type in impossible configurations; ii) small-change recruitment logic to enforce a process has at most one storage server;

2. Test the feature in a real cluster to ensure when storage type is changed: i) no OOM error on a process; ii) SS will be converted to new storage type; iii) storage engine type is changed on storage servers one by one.

[xumengpanda]

Two more considerations: 1) For smaller cluster, for example local cluster on a simple computer for development purpose, we may want to allow multiple SS on the same process to simplify the process; 2) For large cluster, we want to ensure 1 SS per process.

[xumengpanda]

To completely solve this issue, we need:

1. Ensure at most one storage server per process, which requires: i) change simulation test such that it will not change storage engine type in impossible configurations; ii) small-change recruitment logic to enforce a process has at most one storage server;
2. Test the feature in a real cluster to ensure when storage type is changed: i) no OOM error on a process; ii) SS will be converted to new storage type; iii) storage engine type is changed on storage servers one by one.

The item 2 has been tested when we switch storage engine to redwoods.

I'm not sure if the other two considerations are necessary for 7.0 release: 1) For smaller cluster, for example local cluster on a simple computer for development purpose, we may want to allow multiple SS on the same process to simplify the process;

2) For large cluster, we want to ensure 1 SS per process. Ensuring at most one storage server per process, requires: i) change simulation test such that it will not change storage engine type in impossible configurations; ii) small-change recruitment logic to enforce a process has at most one storage server;

[jzhou77]

We have "perpetual storage wiggle" now for storage engine migration.