monitor-cluster with rich dashboard

[RobertTLange]

Currently, we are using a hacky combination of terminaltables and colorclass to print the current resource usage of the cluster. Let's move to rich. It looks beautiful and not too bad in terms of adaptation of our current setup. Furthermore, I would like to replace the tabulate table protocol visualization in load_local_protocol_db. This would reduce our install dependencies and replace terminaltables, colorclass and tabulate by a single package.

Check out this mini-tutorial for dashboards: https://www.willmcgugan.com/blog/tech/post/building-rich-terminal-dashboards/

[RobertTLange]

Here is a little progress report of what I got so far:

Open things to do:

• [x] Add memory/CPU/GPU utilisation for all nodes. Had a quick exchange with Dom and qconf -se <host_id> gives overview of resources for a host. Also qstat -j <job_id> gives all info associated with a specific job. Only problem: I am not sure whether this also works for jobs that are not your own.
• [x] Also replace table in protocol_summary print at experiment startup with rich version.
• [ ] Extend this to Slurm cluster.
• [ ] Extend this to GCP usage.
• [ ] Document process and timeline in a blogpost.

The solution for getting CPU utilisation works via qhost -h <host_name>. The CPU utilisation is given in percent LOAD and can be multiplied with NCPU! Memory on the other hand can be found in MEMTOT and MEMUSE.

[RobertTLange]

There seems to be a python API for monitoring GCP resources: https://googleapis.dev/python/monitoring/latest/query.html Main problem is the lack of nice examples :)

[RobertTLange]

We now have a basic version that works for all three commonly used resources:

• sge-cluster: #36
• local: 591dc14fba8f09c2a903e26509e6ee04efab43c6
• slurm-cluster: e584482e8c0ffc136f9f853a92b95903fe3c41cf

In the future: Look into monitoring GCP as well.