Adds a utility_metric_freshness transform with an example
Motivation and Context
This makes it possible to measure message / data freshness (commonly used to measure data processing systems). Unlike other metric transforms, this emits two metrics with unique attributes to differentiate them in the downstream monitoring platform. A freshness SLO can be calculated by turning these metrics into a percentage (e.g., SLO = success / (success + failure) for a particular pipeline).
It's up to the user to determine when to begin measuring freshness (as determined by the object.source_key timestamp value), but two examples are:
Measure freshness of an individual pipeline component (using the time_now transform)
Measure end-to-end freshness of an entire pipeline (using the timestamp value from the data source)
How Has This Been Tested?
Integration tested using the example config. End-to-end testing will come after this is merged to main (as part of testing the next release).
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
Description
utility_metric_freshnesstransform with an exampleMotivation and Context
This makes it possible to measure message / data freshness (commonly used to measure data processing systems). Unlike other metric transforms, this emits two metrics with unique attributes to differentiate them in the downstream monitoring platform. A freshness SLO can be calculated by turning these metrics into a percentage (e.g.,
SLO = success / (success + failure)for a particular pipeline).It's up to the user to determine when to begin measuring freshness (as determined by the
object.source_keytimestamp value), but two examples are:time_nowtransform)How Has This Been Tested?
Integration tested using the example config. End-to-end testing will come after this is merged to main (as part of testing the next release).
Types of changes
Checklist: