[FEATURE] Export Harvester Performance Metrics

[aiwantaozi]

Export Harvester Performance Metrics

API Calls Metrics

metrics

• HTTP requests, partitioned by status code, method, and host
• HTTP requests laytency, partitioned by status code, method, and host

  solution

  kubernetes already export these metrics, apiserver_request_total, apiserver_request_duration_seconds_bucket etc. example: apiserver_request_total{code="0",component="apiserver",contentType="application/json",endpoint="https",group="harvesterhci.io",instance="172.16.1.31:6443",job="kubernetes",namespace="default",resource="users",scope="cluster",service="kubernetes",verb="WATCH",version="v1beta1"}

Work Queue Metrics

metrics

• depth
• adds_total
• queue_duration_seconds
• work_duration_seconds
• unfinished_work_seconds
• longest_running_processor_seconds
• retries_total

solution

implement interface MetricsProvider, and set SetProvider for workqueue

Reconcile Metrics

metrics

• reconcile_error

  solution

  add metrics in common controller handle, increase the count when error happend. example：sum(rate(rancher_workqueue_adds_total{job=~".harvester."}[5m])) by (instance, name)

Resources

metrics

• cpu
• memory
• network

  solution

  use container_memory_working_set_bytes, container_cpu_usage_seconds_total, container_network_receive_bytes_total

Goroutines, GC Times

metrics

• process collector, including CPU, Memory, file descriptor usage etc.
• go collector, including go runtime metrics, like gc stats etc.

  solution

  prometheus.NewProcessCollector(prometheus.ProcessCollectorOpts{})
  prometheus.NewGoCollector()

[aiwantaozi]

Grafana dashboard:

[w13915984028]

connect with https://github.com/harvester/harvester/issues/2517