TiFlash没有暴露proxy的metrics

[giant-panda666]

Question

Before asking a question, make sure you have:

• Reviewed relevant Kubernetes information: Google your error messages and look at K8s docs.
• Searched open and closed GitHub issues
• Read the documentation:
  • TiDB Operator Readme
  • TiDB Operator Doc tiflash的proxy的metric和自身的metric是分开的，但是只将自身的metric端口暴露了出来，没有将proxy的metric端口暴露出来(无法通过cr配置)。即使暴露出来，因为annotation不一样，prometheus还是无法直接获取metric，建议是使用同一个端口暴露出来。

[csuzhangxc]

We added different annotations for TiFlash and TiFlashProxy, https://github.com/pingcap/tidb-operator/blob/v1.6.0/pkg/manager/member/tiflash_member_manager.go#L525-L526

And added different scrape jobs for them, https://github.com/pingcap/tidb-operator/blob/v1.6.0/pkg/monitor/monitor/template.go#L102-L103

Did this not work in your environment? which version of TiDB Operator are you using?

[giant-panda666]

Yes, TiFlash and TiFlashProxy port added in source code and service, but where is it in Pod?

TiDB Operator version is v1.5.3, and the service created by operator: but the pod created by operator(or statefulset) has no TiFlashProxy metric port：

[csuzhangxc]

there is no need to add the proxy metrics port (20292) to the container.

have you tried to query some metrics for TiFlash-proxy via our TidbMonitor? Are there any metrics that can't be retrieved?

[giant-panda666]

TidbMonitor is not deployed in our cluster, because we use Prometheus to collect all metrics in our cluster as a cloud native standard. Well, thank you for the information that TidbMonitor provides TiFlash-proxy metrics, i can use the relabel_configs of Promethreus to scrape TiFlash-proxy metrics from 20292. But why not add the proxy metrics port (20292) to the container ports of Pod definition as you indeed listen 20292 in the TiFlash process?

[csuzhangxc]

from the code comment of Kubernetes

    // List of ports to expose from the container. Not specifying a port here
    // DOES NOT prevent that port from being exposed. Any port which is
    // listening on the default "0.0.0.0" address inside a container will be
    // accessible from the network.
    // Modifying this array with strategic merge patch may corrupt the data.
    // For more information See https://github.com/kubernetes/kubernetes/issues/108255.
    // Cannot be updated.

add port into container spec is not necessary. and If we add it to container, all existing Pods will be recreated if we upgrade TiDB Operator.

[giant-panda666]

Well, upgrade is truly a thorny problem. Thank you again and last question that why TiFlash uses two ports to expose metrics ?

[csuzhangxc]

Well, upgrade is truly a thorny problem. Thank you again and last question that why TiFlash uses two ports to expose metrics ?

TiFlash itself includes two components, both of which use different config and metrics port

[giant-panda666]

Got it. Thank you for explaining patiently and i will close this issue.