Pipeline Metrics

[dominiklohmann]

We want to implement metrics for pipelines for two purposes: Understanding how they perform (especially when they do not), and getting throughput figures for managed pipelines in the community edition.

To this end, we define two terms:

• Pipeline Ingress is the outbound processing rate of the source operator of a pipeline
• Pipeline Egress is the inbound processing rate of the sink operator of a pipeline

These are the two key metrics that we want to have as KPIs. However, there are also other performance-relevant metrics that we want to collect for every operator in a pipeline:

• Total inbound elements
• Rate of inbound elements per second
• Average number of elements per inbound batch
• Total outbound elements
• Rate of outbound elements per second
• Average number of elements per outbound batch
• Time elapsed since starting the operator
• Time that the operator was scheduled
• Time that the operator spent starting up
• Time that the operator spent running

An element in this context is either a byte or an event, depending on the operator's inbound and outbound element types.

### Definition of Done
- [x] Gather the described metrics
- [x] Design how we render metrics in the frontend
- [x] Ship metrics to the frontend
- [x] Always expose bytes metrics in API endpoint
- [x] Render metrics in the frontend

[dominiklohmann]

In a meeting today with @annadehtiarova and @Dakostu we discussed where we want to go with the design for this.

We agreed on the following: We will split this roadmap item into two: Pipeline Metrics and Operator Metrics.

For Pipeline Metrics we will focus on Pipeline Ingress and Pipeline Egress as total values (e.g., 512,237 events or 94.21 MiB) and average rates (e.g., 42,500 events/s or 10 MiB/s).

For Operator Metrics we want to focus on inbound and outbound metrics per operator, potentially including visualizations with graphs. For that, we want to explore a few possible design options and have a lot of open questions, so that will move back to the Product team first as a new roadmap item.

We also discussed that we want to show metrics also for the Explorer in the context pane. That should be part of the Operator Metrics roadmap item.

@annadehtiarova will create a design that incorporates these metrics into the overview page.

In a follow-up technical discussion with @jachris and @Dakostu we agreed on how to (1) deliver the metrics to the frontend and (2) how to route the collected metrics to that inside Tenzir. The frontend will receive the pipeline metrics as part of the /pipeline/list endpoint.

[annadehtiarova]

Design for pipeline metrics toggles was reviewed here.