Improve components' readiness check mechanisms

[wxiaoke]

Feature Request

Is your feature request related to a problem? Please describe:

Use "kubectl get pod" to check that the status of pd is running, but the process of pd has not yet started

Describe the feature you'd like:

Add pod readiness check to check whether the status of components (PD, TiDB, TiKV) is normal Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

The design of the current operator will seriously mislead users: in fact, the service does not start up, but the status displayed by the get pod is running;

It is strongly recommended to add a pod readiness check

[hanlins]

Thanks @wxiaoke! Rephasing the issue in english for a bit. Basically the pods status are somewhat misleading, take PD pods for example, when a PD pod just come up, by default k8s will consider it as ready as long as the process is up and running, however, PD could be actually syncing metadata with other PD pods which means it cannot serve requests at the time. Need to expose some readiness probe so k8s/tidb-operator can better coordinate the pods and minimize the turbulence on changes (e.g. rolling upgrade).

[mikechengwei]

This is the same problem as #3857 , I will finish it quickly.

[mikechengwei]

/assign

[mikechengwei]

close by #4761 .

[csuzhangxc]

4763