pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.22k stars 493 forks source link

Auto configure pd-server metric-storage #2331

Closed tennix closed 4 years ago

tennix commented 4 years ago

Feature Request

Is your feature request related to a problem? Please describe:

Currenlty, TiDB dashboard will report the following errors if users don't configure metric-store for pd-server.

Error 1105: query metric error: [domain:9009]Prometheus address is not set in PD and etcd, Currently, the PD config `pd-server.metric-storage` value should be prometheus address, please check whether the config value is correct, you can use below sql check the value: select * from information_schema.cluster_config where type='pd' and `key` ='pd-server.metric-storage'; , you can use this shell command to set the config: `curl -X POST -d '{"metric-storage":"http://{PROMETHEUS_ADDRESS}"}' http://{PD_ADDRESS}/pd/api/v1/config`, PROMETHEUS_ADDRESS is the prometheus address, It's used for query metric data; PD_ADDRESS is the HTTP API address of PD server, all PD servers need to set this config. Here is an example: `curl -X POST -d '{"metric-storage":"http://127.0.0.1:9090"}' http://127.0.0.1:2379/pd/api/v1/config

Describe the feature you'd like:

Now tidb-dashbord is included in pd, it requires following configurations:

[pd-server]
## the metric storage is the cluster metric storage. This is use for query metric data.
## Currently we use prometheus as metric storage, we may use PD/TiKV as metric storage later.
## For usability, recommended to temporarily set it to the prometheus address, eg: http://127.0.0.1:9090
metric-storage = ""

Most of the users will use tidb-operator managed monitor, so the prometheus address is known in advance. It would be better if tidb-operator can automatically configure pd if a user does not provide an explicit metric-storage configuration.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

breezewish commented 4 years ago

Hint: in order to be compatible with Dashboard, an etcd key /topology/prometheus must exist (and no need to set metric-storage) containing Prometheus host and port. For the value of the key, see https://github.com/pingcap-incubator/tiup-cluster/pull/295/files