search

dmwm / dbs2go

DBS server written in Go
MIT License
5 stars 4 forks source link

Add runtime metrics to prometheus #82

Open vkuznet opened 2 years ago

vkuznet commented 2 years ago

Following approach of https://github.com/arl/statsviz/tree/1ff44fae9a7555de21fe9ad6fa09787373cf23fa we can add runtime metrics to our metrics endpoint and expose them to Prometheus. For that we should get runtime.MemStats object and convert it to our metrics code base web/metrics.go

The list of supported runtime metrics can be found here

d-ylee commented 1 year ago

For this, are we converting the values in runtime.MemStats, then formatting it in accordance to Prometheus metrics? Would these be steps to add this to our metrics endpoint?

  1. Add the runtime metrics as fields to the Metrics struct: https://github.com/dmwm/dbs2go/blob/bc37f6215560415650b12df197e81874c1bcce91/web/metrics.go#L96

  2. Then add the metrics to the struct: https://github.com/dmwm/dbs2go/blob/bc37f6215560415650b12df197e81874c1bcce91/web/metrics.go#L130

  3. Then format the metrics so that Prometheus would be happy parsing them? https://github.com/dmwm/dbs2go/blob/bc37f6215560415650b12df197e81874c1bcce91/web/metrics.go#L204

vkuznet commented 1 year ago

yes, it is correct procedure.

d-ylee commented 1 year ago

@vkuznet I tried this method using Prometheus' Go Client. I implemented this temporarily on the /runtimemetrics endpoint Would this be something we want to use? https://github.com/d-ylee/dbs2go/commit/0619377ffaaa7545a9258ba8e377ee161a8193ad

vkuznet commented 1 year ago

Dennis, the new code looks solid to me, and since you implemented new metrics on dedicated end-point we can proceed with their deployment and build new dashboard for them. Then, we can compare existing dashboard with new one, and, finally, merge them together. Feel free to merge and proceed with deployment on testbed. But since you put them on dedicated end-point we'll need an additional exporter to scrape them for Prometheus since k8s manifest file only allow single end-point/port to use via annotations, see https://github.com/dmwm/CMSKubernetes/blob/master/kubernetes/cmsweb/services/dbs2go-global-r.yaml#L60-L62

d-ylee commented 1 year ago

@vkuznet Would it be an issue if the metrics scraped have the same name as the original metric? Or should I modify the prefix on the new endpoint?

vkuznet commented 1 year ago

Even though Promeethes us very flexible I think it would be wise to setup originally different prefix. In a long run you want to merge both metrics under single end-point and having single prefix but until everything is tested I do not want to break existing (production) dashboards, e.g. https://monit-grafana.cern.ch/d/kJOvR2h7k/dbs2go?orgId=11