Metrics Source

[dominiklohmann]

Metrics are currently split in two: We have the component-based metrics, and the new performance metrics from pipeline execution. However, accessing metrics currently requires writing them to a file or Unix domain socket first.

This roadmap item is about designing and implementing a source operator that provides metrics as events in real-time.

- [x] Agree on how we route data into this operator.
- [x] Agree on whether we want to keep the old metrics mechanism around.
- [x] Agree on whether we want to be able to access past metrics.
- [x] Implement the operator.
- [x] Store operator metrics.
- [x] Store system health metrics: disk usage, mem usage, cpu usage.

[dominiklohmann]

In a meeting on dashboards today, we said it would be nice to have a mechanism like export for logs (and audit logs), metrics, and diagnostics. We already had a similar idea for the subscribe operator, for which we wanted to re-use the catalog to store the data under different topics, so that export would just become a special case of the subscribe operator that would use an internal topic. We can use a similar internal topic for logs, metrics, and diagnostics.