search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.83k stars 1.58k forks source link

Support configurable metrics aggregations with percentile/histograms transformation #2609

Open szibis opened 4 years ago

szibis commented 4 years ago

For aggregation in for. example statsd would be awesome to have in Vector transformation to do aggregations like min, max, stdev, mean, count, sum on a defined pipe of metrics but what is also needed is to have bucketing to get custom percentiles from a defined list to calculate like 

percentiles = [50,70,90,95,97,98,99]

Also, histogram support with an example for ms buckets in duration calculation.

histogram = [1,10,100,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000]

this will add a set of tags based on histogram per aggregated window of metrics. We may also have the option to choose if we like to add histogram and/or percentiles data as new fields or new tags list of buckets - Some systems better support fields query language based on some better query on tags at the end-user experience.

This is connected with https://github.com/timberio/vector/issues/676 to have defined aggregation windows per this transformation.

I think would be nice to move aggregations as most do from source more behind with more control from Vector. about which metrics we like to aggregate and which not.

For this example, we can have current statsd input with some improvement on supported types - https://github.com/timberio/vector/issues/2603 and move aggregation into transforming part to be more elastic. This may also apply for metrics from logs and more sources.

4wdonny commented 1 year ago

Bumping to see if this is being considered for work anytime soon. We recently tried to migrate our metrics pipeline to Vector from Veneur and realized that Vector is not able to do this at this time, so our distribution metrics aren't showing up in Datadog correctly (no more 95th or 99th percentiles).

bruceg commented 1 year ago

This is not currently being considered for work, but I have bumped it up to see if that might change on the next review.

jszwedko commented 1 year ago

@4wdonny distribution metrics should be sent to Datadog as "sketches". They won't publish percentile metrics, but you can query the percentiles of these sketches (like p50, p99, etc.) without having them previously aggregated. See https://docs.datadoghq.com/metrics/distributions/

jszwedko commented 1 year ago

Hey @szibis !

We were discussing this one today and realized we weren't 100% sure what was being asked for here. Would it be possible to provide some more detail? Is it that you want to pre-aggregate distribution and histogram metrics in Vector into percentiles? Or do you want to turn gauges into histograms? Or something else? It seems like we may want to split this issue up into multiple if there are multiple use-cases.