The web UI has a nice feature that shows commonly-used job metrics directly in the pipeline detail page, including event rate metrics and backpressure metrics. Today, this works like this:
requiring prometheus and pushgateway in order for the web ui metrics to work.
As part of the effort in 0.11 to remove all external dependencies (following on #626), this PR removes the need for prometheus by adding a simple metrics system within the controller.
Now, the controller queries metrics from the workers, computes rates using a exponentially-weighted moving average, stores a limited history (5 minutes), and responds to queries from the API for those metrics.
The web UI has a nice feature that shows commonly-used job metrics directly in the pipeline detail page, including event rate metrics and backpressure metrics. Today, this works like this:
requiring prometheus and pushgateway in order for the web ui metrics to work.
As part of the effort in 0.11 to remove all external dependencies (following on #626), this PR removes the need for prometheus by adding a simple metrics system within the controller.
Now, the controller queries metrics from the workers, computes rates using a exponentially-weighted moving average, stores a limited history (5 minutes), and responds to queries from the API for those metrics.