search

openziti / fablab

The Fabulous Laboratory
Apache License 2.0
11 stars 4 forks source link

Host Metrics #24

Closed michaelquigley closed 4 years ago

michaelquigley commented 4 years ago

Implement a simple host metrics probe (CPU, memory), and integrate it into the zitilab/characterization/ziti (#17) model.

michaelquigley commented 4 years ago

The most universally reliable way to implement this is probably going to be to use sar via remote puppetry (remote ssh):

$ sar -u -r -q 1 5
Linux 5.4.12-100.fc30.x86_64 (zero-lx.quigley.com)  01/21/2020  _x86_64_    (6 CPU)

07:03:20 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:03:21 PM     all      5.26      0.00      5.60      0.00      0.00     89.13

07:03:20 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:03:21 PM    322248   3985760   7799624     96.03    258932   3653992   7357160     49.60   4638456   2128528       676

07:03:20 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:03:21 PM         1       929      0.83      0.72      0.43         0

07:03:21 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:03:22 PM     all      3.41      0.00      4.94      0.00      0.00     91.65

07:03:21 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:03:22 PM    322012   3985532   7799860     96.04    258940   3653992   7357160     49.60   4638868   2128528       736

07:03:21 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:03:22 PM         0       929      0.76      0.71      0.42         0

07:03:22 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:03:23 PM     all      3.40      0.00      4.08      0.00      0.00     92.52

07:03:22 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:03:23 PM    321524   3985308   7800348     96.04    258940   3654248   7357160     49.60   4640780   2128784       992

07:03:22 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:03:23 PM         2       929      0.76      0.71      0.42         0

07:03:23 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:03:24 PM     all      5.43      0.00      7.30      0.00      0.00     87.27

07:03:23 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:03:24 PM    311608   3975432   7810264     96.16    258940   3654284   7357160     49.60   4651840   2128812      1064

07:03:23 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:03:24 PM         0       929      0.76      0.71      0.42         0

07:03:24 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:03:25 PM     all      3.58      0.00      3.75      0.00      0.00     92.67

07:03:24 PM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:03:25 PM    306876   3970988   7814996     96.22    258940   3659956   7362440     49.63   4649452   2134424      1356

07:03:24 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
07:03:25 PM         4       928      0.76      0.71      0.42         0

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      4.22      0.00      5.14      0.00      0.00     90.65

Average:    kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
Average:       316854   3980604   7805018     96.10    258938   3655294   7358216     49.61   4643879   2129815       965

Average:      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
Average:            1       929      0.77      0.71      0.42         0

We'll probably want to start a poller for each host in the model, and then poll using sar and them consume those snapshots. Will require writing a basic output parser.

michaelquigley commented 4 years ago

Moving to "closer" model used by the tcpdump implementation.