Custom Metrics Support for Syncer

[tpouyer]

Is your feature request related to a problem? Please describe. As an administrator of clusters "connected" to KCP I want to monitor the health and performance of the syncer running on my physical clusters.

Describe the solution you'd like Cluster administrators already have tools and utilities to monitor cluster metrics such as pod restarts, memory, CPU utilization, etc... however many workloads support exporting their own custom metrics that can be gathered by these same administration monitoring tools. It would be good for the syncer (as it can be considered just another workload on a physical cluster) to export metrics that are specific to it's health. This is not an exhaustive list but some metrics that come to mind:

• syncer version
• connected kcp version
• heart beat/last timestamp of message sent or received from kcp
• number of resources synced since last reported
• time to sync resources since last reported
• status of syncer reported to kcp with timestamp
• cpu/memory capacity reported to kcp

Describe alternatives you've considered Currently monitoring CPU/Memory consumption of syncer on physical clusters

Additional context

[skonto]

I could take a look on this one if scheduled. Some steps required: a) define the metrics in a proper format eg. labels, value type etc, following some standard eg. OpenTelemetry. b) deal with string metrics (not directly supported in some backend systems) eg. version, could be just labels. c) use a proper client library to expose the metrics on a port. eg openetelemetry-go.

[mjudeikis]

/transfer-issue contrib-tmc