Adding Pod restart window config

[luohao]

Feature Request

Is your feature request related to a problem? Please describe:

Currently tidb-operator supports maintenance window via updating the pause flag in TidbCluster CR(doc). In our case we have a TiDB cluster that spans across multiple k8s cluster(i.e., deployment managed by multiple operator), and we want to ensure Pod restarts by pod controller (triggered by annotation) in different clusters don't overlap. It's possible to achieve this by turning on/off the pause flag, but we ant to avoid constantly updating the TidbCluster CR.

Describe the feature you'd like:

1. Adding a config in tidb-operator that defines a maintenance window where all reconciles are paused. For instance we can use a CRON schedule+duration to define the window. The following example configures an operation window from 0-8 every day.

    --maintenance-schedule="0 0 * * *"
    --maintenance-duration="8h"

2. The pod controller should skip Pod restarts if it's not in operation window. For example, in the sync loop of pod controller

func (c *PodController) sync(key string) (reconcile.Result, error) {
    if c.inOperationWindow(time.Now().UTC()) {
        return reconcile.Result{}, nil
    }
...

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[csuzhangxc]

Is it possible to scale tidb-operator to 0 with a cron job?