Optimization on Kubernetes

[DanielZhangQD]

Though we already support deploying TiFlash on Kubernetes with TiDB Operator, there are still some optimizations that need to be done to run TiFlash on Kubernetes more gracefully:

1. The default configurations handling. Currently, there are many configuration parameters that need to be set default values explicitly by both TiDB Operator and TiUP (check this big file to set default config), which is tedious and may not sync up with the behavior change in TiFlash, for example, someday TiFlash may change the default value of one parameter but its default value is not updated in TiDB Operator or TiUP in time.
2. Many processes are combined in one container This is not a good practice to run applications in the container, usually, one process per container. Currently, there are 3 processes and 4 log files in the TiFlash container, which brings the following restrictions:

• The health check of Kubernetes cannot work as expected because it can only check one process, although we can write a script to check all of the 3 processes, it's not the best practice anyway.
• The 4 log files are written to the disk (we can print the proxy logs to stderr and we still have 3), however, usually, the applications are printed to stdout/stderr and users will deploy central log service, e.g. EFK/ELK, to collect all of the logs in the Kubernetes cluster, and they only collect the logs from stdout/stderr by default so the logs of TiFlash will not be collected, we now have to add 4 sidecar containers to tail the logs to work around this issue.

Is it possible to deploy the 3 processes in 3 containers in one Pod?

[lidezhu]

1. Cluster manager is a regularly started process, so it is not necessary to check it's status. And proxy is just a thread. So it's ok to run tiflash in one container.
2. server.log and proxy.log are the most important log that we needed. Other logs is not necessary to be colloected.