[Feature] AutoScaling For TidbCluster

[Yisaer]

Description

Describe the feature you'd like: TiDB-Operator is going to support Auto-scaling feature for TiKV and TiDB in TidbCluster.

Auto-scaling would help the Operator users to auto-scale out/in for the TiKV and TiDB for the TiDBCluster by the metrics / value / resources which the users could provide. This issue is to discuss and track the whole process of the Auto-scaling design and realization.

First, I think there are some prerequisites to the Auto-Scaling for TidbClusters:

• The Customized Ability for the Auto-scaling Algorithm. (must to have) TidbCluster is sensitive to the scaling as the distributed databases, we should have the ability to control the whole auto-scaling process.
• The Metrics configurations should be extendable. (must to have) Currently, the Auto-scaling need the TidbCluster metrics info to decide the recommended numbers of TiKV and TiDB. In the future, the Platform information (Kubernetes) or the external global metrics/values are also necessary.
• The Interval duration Control of the Auto-scaling should be Cluster Level. (nice to have) Operator could manager several clusters in one Kubernetes. We should provide the cluster level control ability and Interval duration is also important to avoid performance jitter.

Auto-Scaling Design To support this feature and meet the prerequisites, the Auto-scaling is designed to create one new API (TidbClusterAutoScaler) and one new Controller AutoScaler Controller.

TidbClusterAutoScaler is kind of like HPA. The Operator users could use it to auto-scale in/out the TidbCluster by their own demands configured in the TidbClusterAutoScaler Spec.

The AutoScaler Controller would watch the TidbClusterAutoScaler and reconcile it to adjust the replicas in TidbCluster.

Category

Auto-Scaling

TODO List

• P0
  • [x] Update CRD for tidbautoscaler #3156
  • [x] Defaulting and validation for tidbautoscaler CR #3157
  • [x] sync tidbautoscaler when spec.xxxx.external is configured #3158
  • [ ] Sync tidbautoscaler with PD API #3159

Workload Estimation (P0 features)

45

Time

GanttStart: 2020-07-13 GanttDue: 2020-09-30

Documentations

Project

[Yisaer]

We welcome everyone to help to realize the Auto-scaling feature in Operator by discuss/suggest/code review/pull request/ etc.

[Yisaer]

Currently, the auto-scaling is under alpha feature. It only provides the ability below:

1. basic auto-scaling Availability
2. basic guarantee to avoid jitter during auto-scaling.

To achieve Production ready:

1. control the scaling step for auto-scaling (need discussed)
2. record the consecutive count timestamp in auto-scaling (need discussed)
3. fetch store info from pdapi .
4. skip the consecutive count control if the auto-scaling results are the same between 2 times. ( need discussed)
5. add timeout for prometheus query.
6. make filterTidbInstance compatible with tidb failover
7. noise reduction.

[Yisaer]

Here describes How the auto-scaling algorithm work by average CPU load.

1722

[Yisaer]

After #1731 merged, we would have auto-scaling ability based by cpu load feature under alpha stage. Currently, there are still plenty of jobs to do:

1. Add Syncing TidbClusterAutoCluster Status
2. Add proper events and logs
3. Design for noise reduction for auto-scaling process
4. Support specific pd label for the auto-scaling out tikv instances.
5. Unit test and e2e test for auto-scaling
6. Support TidbMonitor

[Yisaer]

There are several good first issue about auto-scaling`, we are welcome the newcomers to join the contribution by assign these tasks.

Ref:

1751

1752

1753

[Yisaer]

Syncing the replicas between online configuration and local configuration is always the problem after we use autoscaler ( or hpa）, this issue request the new feature to solve this problem. https://github.com/pingcap/tidb-operator/issues/1818

[Yisaer]

To improve the user-experience, we should enhance the information by executing

kubectl get tidbclusterautoscaler

Ref: https://github.com/pingcap/tidb-operator/issues/1820

[Yisaer]

Currently, we have released Auto-scaling as an alpha feature in operator 1.1 version which based on the cpu load. After that, we would start to focus on the following 3 things:

1. support more kinds of metrics in auto-scaling
2. The noise reduction for auto-scaling
3. For tikv and tidb auto-scaling, we would try to use heterogeneous design instead of mutation webhook.

The e2e test should also be completed.

[Yisaer]

We are happy to announce that the auto-scaling is going to have the external strategy ability that exposes the http interface to let community user could use their own auto-scaling strategy (like predicting strategy by AI) to affect to tidbcluster auto-scaling.

For more detail, see: https://github.com/pingcap/tidb-operator/pull/2279

[DanielZhangQD]

• UCP:
  • P1:
  • noise reduction https://github.com/pingcap/tidb-operator/pull/2307
  • noise reduction for tidb
  • control the scaling step for auto-scaling https://github.com/pingcap/tidb-operator/issues/2372
• Non-UCP:
  • The replicas updated by auto-scaler may be overwritten by applying local yaml https://github.com/pingcap/tidb-operator/issues/1818
  • fetch store info from pdapi
  • make filterTidbInstance compatible with tidb failover
  • support more kinds of metrics in auto-scaling
  • For tikv and tidb auto-scaling, we would try to use heterogeneous design instead of mutation webhook.