search

Altinity / clickhouse-operator

Altinity Kubernetes Operator for ClickHouse creates, configures and manages ClickHouse® clusters running on Kubernetes
https://altinity.com
Apache License 2.0
1.91k stars 461 forks source link

ServiceMonitor also selects Clickhouse Nodes #1350

Open derbauer97 opened 8 months ago

derbauer97 commented 8 months ago

Hey,

i noticed when i use the ServiceMonitor in the HelmChart also the clickhouse nodes are selected. So Prometheus tries to scrape them on Port 8888 and Port 9999 which leads to an connection refused.

As far as i understand it is not intentend to scrape the Clickhouse Nodes directly but the clickhouse-operator which provides the metrics. My dashboard is already filled with the correct metrics but i get TargetDown Alerts which i want to fix.

I think it is because the Clickhouse Nodes is:

spec:
  endpoints:
  - port: clickhouse-metrics
  - port: operator-metrics
  selector:
    matchLabels:
      app.kubernetes.io/instance: clickhouse
      app.kubernetes.io/name: altinity-clickhouse-operator

i think the issue is that the Operator and the Clickhouse Services both have the same labels and therefore get picked up by the ServiceMonitor.

Slach commented 8 months ago

how exactly do you install your kind: ClickHouseInstallation manifest?

did you use values.yaml something like that?

serviceMonitor:
  enabled: true
additionalResources:
 - |
   apiVersion: clickhouse.altinity.com/v1
   kind: ClickHouseInstallation
   metatada:
    ...
   spec:   
    ...
derbauer97 commented 8 months ago

Yes i pass it via additionalResources.

Slach commented 8 months ago

In clickhouse-operator deployment, 8888 port used for export metrics related to clickhouse-server and these metrics exported by metrics-exporter container which scrape your clickhouse-server nodes via SQL HTTP queries, and 9999 port contains metrics related to clickhouse-operator itself, which exported by clickhouse-operator container

Why do you ensure your clickhouse-server related pods contains the same labels which present in matchLabels in your ServiceMonitor?

derbauer97 commented 8 months ago

I do not pass any additional Labels i think the labels are passed via helm by default but i am not 100% Sure. So since the ClickhouseInstalltion is part of the Clickhouse-Operator the labels are passed automatically. So i think when i is suggested to pass Clickhouse Installtion via .additionalResources the Clickhouse Metrics Service should have another Selector than only the helm default labels?

https://github.com/Altinity/clickhouse-operator/blob/master/deploy/helm/templates/_helpers.tpl#L49

We could include an existing label like: clickhouse.altinity.com/chop=0.23.2 which is unique for the clickhouse-operator pod or implement an completly new label which not relys on the helm defaults.