Job metrics

[bloodearnest]

We have a simple emergency metrics that writes to local sqlite db.

We want a metrics system that:

• has more robust collection that current cli output parsing
• ships metrics up to job-server
• data easily accessible to job-server to include in UI
• admin UI/query engine for debugging

Proposed MVP

• use psutil to collect data more robustly
• write to backend local sqlite db (naive proposal: 1 table for host metrics, 1 for job)
• replicate to job-server (e.g. https://litestream.io/)
• expose per-backend RO datasette instance for admins
• job-server can open RO connection to local sqlite db

[evansd]

Broadly speaking this sounds great. I love the idea of just dumping to sqlite and then using litestream to stream it out.

We'll need something docker-aware though to track per-container stats, volume usage etc. I'm not use if psutil supports that out of the box, but it looks extensible in any case.