[Metrics] Extend metrics to measure physical performance stats

ucbrise / clipper

A low-latency prediction-serving system

http://clipper.ai

Apache License 2.0

1.4k stars 280 forks source link

[Metrics] Extend metrics to measure physical performance stats #421

Open dcrankshaw opened 6 years ago

dcrankshaw commented 6 years ago

It would be great to extend our monitoring infrastructure to measure more physical performance. To start with, @blackhat06 suggested tracking the following resource metrics:

[ ] Disk IO: % time that device was busy
[ ] Memory: % of total memory capacity in use
[ ] CPU Utilization: 5,10,15 min
[ ] Memory Utilization: Breakdown by memory
[ ] % volume usage: Disk all mounted
[ ] Bits IN/Out (ethernet)
[ ] Volume I/O
[ ] Process count / running/blocked

simon-mo commented 6 years ago

Prometheus can track these with node exporter https://github.com/prometheus/node_exporter/blob/master/README.md

For Kubernetes we can just scrape kube-api-server/metrics. Kubernetes expose Prometheus metrics there

simon-mo commented 6 years ago

Update:

For Docker, we can safely assume user only has on node so we can just run a node exporter at startup.
Kubernetes's API service does expose metrics but the metrics are about the api server requests itself and etcd usage. We should use a DaemonSet. It will deploy a prometheus node exporter to each node (https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). In fact this is recommended practice by Kubernetes.

dcrankshaw commented 6 years ago

Awesome. cc @blackhat06

rkooo567 commented 5 years ago

@simon-mo Is this handled?