Open bboreham opened 5 years ago
Note to have process-level metrics the target system would have to run something like https://github.com/ncabatoff/process-exporter
@bboreham Does this issue assume the user of Scope will have Prometheus installed and accessible on their side or would we somehow install it ourselves with Scope? Also, if Prometheus wasn't there, should the metrics be absent or shall we use probe metrics as a fallback?
I think we would make this optional, e.g. a command-line flag. So if the user knows they have Prometheus metrics available they can set the flag, and give a URL to reach Prometheus, and if not then Scope will collect and display metrics the way it does today.
Make the sparkline charts in the UI source their data from a Prometheus instance that has suitable labels*.
Also, if possible, make the "glass half full" overlay on nodes come from Prometheus too.
Benefit will be that probe can stop gathering this data, which is cpu-intensive, and stop uploading it, which is network-intensive. I expect it will make delta reports (#3677) much smaller.
By default Scope shows 15 seconds of data at 1-second resolution; generally Prometheus is run with 15-second or 1-minute resolution so the Scope charts would have to show a much bigger time window. I think this is a benefit, since the 15-second charts either dodge around too fast to gather much meaning, or are completely flat.
* probably "suitable labels" means this will be Kubernetes-only (at least as an MVP).
Related: #2664, #1619, #2832