Add support to customize spare replicas during VolumeReplace (#5666)

This is an automated cherry-pick of #5666

What problem does this PR solve?

Currently the VolumeReplace feature assumes a default spare replica of 1 for PD & TiKV. Sometimes it is useful to change this number. Setting a larger spare replica number helps for faster tikv region transfer In multi-az setups, it is possible to get zone skewness with only a single replica, which will cause issues during scale down of spare replica.

Example multi-az senario: 3 zones , 6 replicas, initial zones per replica: 1,2,3,1,2,3 After adding 1 spare replica: 1,2,3,1,2,3,1 Now replace will start replacing from start, but now the new disks can get different zones, example: after replacing first disk it could now become 2,2,3,1,2,3,1 After this if we scale down the spare replica (last one), it will become 2,2,3,1,2,3 which is zone-skewed and can be blocked depending on configured topology constraints.

In this case setting spareReplicas to 3 will avoid this issue

What is changed and how does it work?

Add a config option in TidbCluster spec, inside TiKVSpec and PDSpec to configure spare replicas

Code changes

[X] Has Go code change
[ ] Has CI related scripts change

Tests

Re ran the manual test described in : https://github.com/pingcap/tidb-operator/pull/5150 Ran once as it is, and observed default spare replica of 1 was used. Ran once by setting tikv spare replicas to 3, and observed 3 spare replicas was used

[ ] Unit test
[ ] E2E test
[ ] Manual test
[ ] No code

Side effects

[ ] Breaking backward compatibility
[ ] Other side effects:

Related changes

[X] Need to cherry-pick to the release branch
[ ] Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

VolumeReplace feature now supports customizing the number of spare replicas used during TiKV or PD disk replacements inside the TidbCluster spec.

pingcap / tidb-operator