search

datasweet / kibana-datasweet-formula

This Kibana plugin allows calculated metrics on any standard kibana visualizations.
Apache License 2.0
121 stars 34 forks source link

Filter ratio? #53

Closed ghost closed 5 years ago

ghost commented 5 years ago

I'd like to do formulas based on filtered counts, is there a way to do this?

I have some logs, and I want to get:

( count of log.status_code > 500 ) / (count)

It doesn't appear there is a way to make an aggregation with a filter without filtering the whole series, is there any way to do this with the plugin?

lfroment-datasweet commented 5 years ago

Hi, parents pipeline aggs might do the trick for you but I am not sure I understand what you really try to do. Do you have an example screenshot? You want to filter out log.status_code>500?

ghost commented 5 years ago

I have a set of http logs. Assume I have fields "logs.host" and "logs.status_code" and of course a timestamp.

X axis: date histogram Y axis: a line for each logs.host representing percentage of status_code 500-599.

Viewing a line of count of errors is easy, but doing it as a percentage of all records for logs.host is not.

I was hoping that the datasweet formula would make this possible, but the underlying difficulty is that I cannot get an agg for both a count and a filtered count at the same time.

The closest workaround I can think of is to create a scripted field "error" that gets set to "0" if it's not a 5xx, and "1" if it is, then I can do a agg of sum of errors. This is clunky but could work.

lfroment-datasweet commented 5 years ago

OK. Sibling pipeline should do it to prepare your filtered metric . like this image

ghost commented 5 years ago

Hm, that's closer and is promising. I need the filter to be logs.status_code: [500-599] (doesn't seem to work for me), then I will break the Y axis aggregation into buckets based on hostname (that part works fine).

ghost commented 5 years ago

Figured it out, I had missed a capital letter :) I use a Sum Bucket Range to capture 500-599, and I think it's giving me what I want.

Thanks for the help, and thanks for the plugin!