Average _count per interval (eg minute) metric

[n3ziniuka5]

I have an index in which every document represents a request. I want a metric that would show requests per minute to measure how busy the site is.

Currently I have a count metric with query @timestamp: [now-1m TO now] which shows the amount of requests received during the last minute.

However, I want to use Kibana's built in duration(top-right corner). So if I select Last 15 minutes, it should get a count of documents indexed during the past 15 minutes and divide that by 15. Similarly, if I select Last 1 hour, it should divide it by 60.

Is it possible to accomplish that with the current stable version of Kibana? If not, do you think it would be a good feature to add?

[johnkors]

aaaaand +1

[Xeteskian]

+1

[Valian]

+1

[sio4]

+1

[timofeic]

+1

[damianesteves]

This would be great, I'm currently trying to get metrics for average daily users on my app, by using the unique count of User IDs active per day, but there seems to be no way to get the average number of records by time frame. It seems all the tools are there to make this happen. I can make a histogram that returns the number of records by day, and this produces a nice little chart with daily values, now we just need to automate the sum and division part to get the avg.

[Funbit]

Can't believe it's not done yet, +1

[andreivirtosu]

+1

[jcredland]

+2

[aakil786]

+1

[dkavraal]

+1

[DeeeFOX]

I solved it by this:

{
  "aggs": {
    "time_unit": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "5h",
        "time_zone": "+08:00"
      },
      "aggs": {
        "total_count": {
          "value_count": {
            "field": "response.code"
          }
        },
        "count_pers": {
          "bucket_script": {
            "buckets_path": {
              "total": "total_count"
            },
            "script": {
              "inline": "total/ interval",
              "params": {
                "interval": 18000
              }
            }
          }
        }
      }
    }
  }
}

It will pre adjust two interval: aggs.time_unit.date_histograme.interval and aggs.aggs.counter_pers.script.params by codes like:

swich(unit) {
    case 's': return val
    case 'm': return 60 * val
    case 'h': return 3600 * val
    ....
}

[skinger]

+1

[danielzheng]

+1

[Quezion]

+1

[michaeldav]

+1

[loekvangool]

+1

[axsapronov]

+1

[cgough]

+1

[hendryanw]

+1

[mikhail-gaponov]

+1

[kapowkimc]

+1

[bqk-]

+1

[jethroguce]

👍

[winnowsteven]

+1 Am just wondering if this is something that will ever get done, given that the thread has been open since Aug 12, 2015. Or, has there been some other workaround that I am not aware of?

[ppisljar]

this should be somehow possible:

so add a date histogram, and for interval select minute. thats it.

if you zoom out to a long time range (lets say one year) you wont see a bar for every minute, but it will still represent per minute value. So lets say that you get one bar per day, it will still show you your per minute count (averaged)

[guomo]

Is there a question about if the community wants this feature or not? Seems like there are loads of upvotes and +1. I spent an hour trying to make this work until I finally came across this feature request. I surprised it wasn't already part of it given Splunk does it rather handily.

[ppisljar]

@guomo as i mentioned in the comment above this is actually working,but i agree its not very clear whats happening ....

[bharat129]

Is there any clear solution for creating graph/chart for Total Transactions/sec (In aggregated way for all transactions), similar to graph in JMeter, LoadRunner or other performance testing tools? Please don't mix it with chart which shows the count for a particular instant.

I could not implement if there any solution above.

[loekvangool]

@bharat129 you're probably looking for Timelion

[Sjaak01]

Is there a fix for this issue yet? I want to graph bandwidth as well.

It's pretty easy to do by taking the sum and dividing it by 60 with a one minute interval but that isn't usable long term because it will create too many buckets.

Timelion's scale.interval doesn't create the correct graph when zoomed up.

It is a bit frustrating to see that after people asking for a (imho basic) feature like this for 2 years there still doesn't appear to be a real solution.

Logstash comes with a netflow collector, you'd expect Kibana being able to do something useful with that data ;)

[nreese]

Looks like this thread really covers two separate enhancement requests. 1) An update to the Metric Visualization that displays the metric value as a ratio (metric value divided by the time picker duration and specified interval). For example, the Metric Visualization is displaying the Sum Metric Aggregation on the field "bytes". Under Options, the user has configured an interval of "1 minute". The time picker duration is 15 minutes. The displayed metric value is sum/15 and the default label is "(sum of bytes) per minute". This sounds like the original intent of @n3ziniuka5's request and @GrahamHannington use case. 2) An update to the Date Histogram bucket aggregation that allows for setting a metric interval that uses a scripted metric to turn the metric value for each bucket into a ratio (metric value divided by the bucket size and interval). This covers the functionality specified by @DeeeFOX and @pagenbag.

[trevan]

@nreese, I think #1 can already be done using the "Average Bucket" aggregation. You can pick a sub bucket aggregation of "Date Histogram" with an interval of "1 minute" and then a sub metric aggregation of "Sum" and field "bytes".

It does like there is a bug with that, though. It isn't handling the scaling correctly when the time range is too big.

2 also might be possible with having an "Average Bucket" metric aggregation and then a normal "Date Histogram" bucket. So the metric would be an average for the smaller timespan but graphed using the larger timespan.

[nreese]

Thanks @trevan for pointing out Average Bucket sibling pipeline aggregation that was supported in Kibana 5.3. Looks like that solves both of these use cases.

1)

2)

[tomryanx]

I don't think the second example proposed by @nreese really works as intended, due to scaling of the time period withing the date historgram on the Y-axis.

Shouldn't this be done in Kibana, rather than in ES itself?

I'm going to call X-interval the date histogram interval on the X-axis, and Y-interval the date histogram interval in the Y-axis subagg.

The proposed behaviour does something like this: for each X-interval, get all Y-interval values from ES, then average them to present a single number (average of all per-Y-interval values) for the X-interval. If there would be too many buckets created for the X-interval (ie X-interval * Y-interval is too large), autoscale Y-interval to reduce the number of buckets.

Whereas the desired behaviour would be: get a single number for each X-interval, then - without performing another date histogram agg on the X-interval buckets, divide the figure by Y-interval.

For example, as in the second gif above: X-interval is 1m, Y-interval is 1s, full period is 15m.

Not so good: 15 buckets for the X-interval is fine, but 60 * 15 = 900 is too many, so Y-interval is scaled to 5s. The resulting numbers show "average count per 5 seconds", which is calculated as something like "average of per-5s-average" for each X bucket.

Better: divide the total for the X-interval by the Y-interval, producing a desired-accuracy average without relying on ES further nor allowing the too many buckets issue to arise.

[SolomonShorser-OICR]

+1 I'd like to be able to show avg. number of requests per hour of the day, and then if possible compare year-over-year (or also do month-by-month comparisons).