Custom Metrics for Pipelines and Filters - Githubissues

logbee / keyscore

Apache License 2.0

3 stars 0 forks source link

Custom Metrics for Pipelines and Filters #72

Open EndallBatan93 opened 5 years ago

EndallBatan93 commented 5 years ago

To get better insights of a pipeline and their filters we need an agnostic concept to provide custom metrics.

[x] reuse datamodel(Fields and Values/ Metadata and Labels)
[ ] ~~displaying them in an additional tab in configurator component~~
[x] reuse translation concept
[x] separate API for aquiring Metric for a specific filter
[ ] ~~angular charts for visualisation (agnostic chart types)~~
[ ] ~~d3 charts~~
[x] enhance Descriptor API to define metrics
[x] enhance valves with common metric methods such as: throughput time, number of datasets etc...
[x] enhance some filters as example. eg. GroupingLogic etc.

kKdH commented 5 years ago

Started to implement mechanisms to gather metrics from sources, sinks, filters etc.

Due to the planned integration with Prometheus. Metrics in KS are similar to Metrics in Prometheus.

There is a first draft of the enhanced Descriptor API and Stages do now have an extra function which can be implemented by a block-developer to offer custom metrics.

mlandth commented 5 years ago

e6b7c06ad1ae3385d2db4ab7ea7eff7fe596e383: Implemented some standard-metrics for the valves.

mlandth commented 5 years ago

ff5f354e9293f17c46d04ac71a763f30dac4f84f Started to implement the scrape functions on the agent side. Also updated the Metrics with Labels.

mlandth commented 5 years ago

382c223040365684ded8a5ca243562fbd10f7076: Added a Integration-Test for the Metrics of a single Pipeline.

mlandth commented 5 years ago

b6ce80c066e65a24e7cf0e5dc77e164138ffed8c: First approach of a MetricsManager.

mlandth commented 5 years ago

4ad81ddada6563c8ec7c65fa5dbf5ab6173956c1: Using an Ehcache now for storing metrics.

mlandth commented 5 years ago

120c16c75805af2e5fae9c17552cd5920b4d82fc: Finished the first approach the MetricsManager and the MetricsCache.

mlandth commented 5 years ago

8b1b1adf208fa1089ca398c82aec21f8b9b5c5f8: Implemented first Metrics for Filter (GroupingLogic)

mlandth commented 5 years ago

15272cb0977ff59562ce45a2b3f1a92e8105fe00: Implemented Metrics for KafkaSource and KafkaSink.

mlandth commented 5 years ago

2bdbfad01186eeefbd33cd2b4bab740b6e27b054: Time-based Metrics API.

mlandth commented 5 years ago

a65aaf82f645f58fafee9a914782d5d3299092d9: Refactored metrics API

mlandth commented 5 years ago

e5b21e59c921e10afd505fee137b8ced130fc991: Updated MetricSource.

mlandth commented 5 years ago

d256d1ec18accb874b1d255e07492fa03d548b5e: Enhanced metrics with TimeDelta

mlandth commented 5 years ago

76cca3646beae2477e24f4967d1975f2d296909f: Updated MetricSource.