metrics support

[zqzten]

In production we want to collect some key metrics of kine for monitoring and alerting, such as the DBStats of Go SQL backend used by kine, SQL operation times and errors etc. Would it be applicable that we add some prometheus metrics for them?

For library mode, we can add an injectable registerer in endpoint config and for stand-alone, we can setup our own metrics handler like etcd.

[brandond]

I think this might already be covered by https://github.com/k3s-io/kine/pull/104?

this PR adds a Maintenance service that only supports Status, populating the DbSize field if the driver has a query to report this data. Currently sqlite, mysql, and postgres all support this; cockroachdb does not. This field is periodically collected by the Kubernetes apiserver and copied into the etcd_db_total_size_in_bytes metric.

Just pull the etcd_db_total_size_in_bytes metric from your apiserver

For other metrics, I'm curious what you might expose and how. Kine is most frequently used in an embedded capacity where it only listens on a Unix socket, which isn't easy to scrape directly.

[zqzten]

Our use case might be a little different from the most common, we integrate kine (as a library) with our custom apiserver so that it can directly call kine's endpoint.Listen to use SQL servers as etcd. In such use case, we can directly pass the apiserver's prometheus registerer to kine in endpoint config and let kine register its metrics to it. For standalone usage, I'm not sure what will be the best practice (since the use case is variable), a naive one may be just setting up a simple http prometheus metrics handler whose port can be configured by cmd flag.

For specific metrics, currently I want to expose below metrics:

• SQL Total & Errors: Counter metrics which hold the total number of SQL operations and failed SQL operations. This is helpful for quickly tracking and alerting local SQL errors which have a variant of causes (network, db, etc.) thus not easy to monitor.
• SQL Time: Histogram metric which keeps track of the duration of SQL operations. This duration is measured locally and can be used as a supplement to the SQL exec time measurement of db.
• Compact Errors: Counter metrics which hold the total number of compact errors. Helpful for alerting when continuous compact errors occur which can lead to uncontrollable growth of db size.
• DBStats metrics of sql.DB used by kine.

[zqzten]

@brandond Any thoughts on this? Do you think these metrics are nice to have?

[brandond]

If you'd use them, then go for it! Love to see the PR. Let's just make sure that the metrics extension points don't add any additional overhead if metrics aren't being collected? Shouldn't be a problem I don't think.

[zqzten]

/assign