elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.77k stars 8.17k forks source link

Wrong value in indexing rate with 7.10.0 #84221

Open fredbcode opened 3 years ago

fredbcode commented 3 years ago

Hello,

Elastic : 7.10.0 Kibana: 7.10.0

Ubuntu 20.04

Right after migration to 7.10.0, the indexing rate value appears very low in Kibana,

Extract about only one index:

image

Kibana index rate:

image

I also checked by api this index, with 10 seconds as range the result is 17216,5 by second

With previous versions indexing rates value was more or less 10 000 r/s

elasticmachine commented 3 years ago

Pinging @elastic/stack-monitoring (Team:Monitoring)

chrisronline commented 3 years ago

Hi @fredbcode ,

Thanks for filing an issue!

I also checked by api this index,

What do you mean by this?

The chart expresses a rate, which means we don't chart raw values, but rather the change over time.

fredbcode commented 3 years ago

What do you mean by this?

The chart expresses a rate, which means we don't chart raw values, but rather the change over time.

Yes, I collected every 10 seconds the total number of documents being indexed, I subtracted the latest value with the previous and divided the result by 10

I know it's an approximate way, but the result difference with kibana is huge, very huge

chrisronline commented 3 years ago

This is the kind of query we run to calculate the values for the chart:

POST .monitoring-es-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "cluster_uuid": "<insert_cluster_uuid>"
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": "now-1h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "check": {
      "date_histogram": {
        "field": "timestamp",
        "fixed_interval": "30s"
      },
      "aggs": {
        "primaries_raw": {
          "max": {
            "field": "indices_stats._all.primaries.indexing.index_total"
          }
        },
        "primaries": {
          "derivative": {
            "buckets_path": "primaries_raw",
            "gap_policy": "skip",
            "unit": "1s"
          }
        },
        "total_raw": {
          "max": {
            "field": "indices_stats._all.total.indexing.index_total"
          }
        },
        "total": {
          "derivative": {
            "buckets_path": "total_raw",
            "gap_policy": "skip",
            "unit": "1s"
          }
        }
      }
    }
  }
}

Try running it and seeing if the values match up to what you see in the UI.

fredbcode commented 3 years ago
{
  "took" : 317,
  "timed_out" : false,
  "_shards" : {
    "total" : 7,
    "successful" : 7,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "check" : {
      "buckets" : [
        {
          "key_as_string" : "2020-12-01T07:40:30.000Z",
          "key" : 1606808430000,
          "doc_count" : 3999,
          "total_raw" : {
            "value" : 4382001.0
          },
          "primaries_raw" : {
            "value" : 2783611.0
          }
        },
        {
          "key_as_string" : "2020-12-01T07:41:00.000Z",
          "key" : 1606808460000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 4402219.0
          },
          "primaries_raw" : {
            "value" : 2793720.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:41:30.000Z",
          "key" : 1606808490000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4422568.0
          },
          "primaries_raw" : {
            "value" : 2803898.0
          },
          "primaries" : {
            "value" : 10178.0,
            "normalized_value" : 339.26666666666665
          },
          "total" : {
            "value" : 20349.0,
            "normalized_value" : 678.3
          }
        },
        {
          "key_as_string" : "2020-12-01T07:42:00.000Z",
          "key" : 1606808520000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4442783.0
          },
          "primaries_raw" : {
            "value" : 2814002.0
          },
          "primaries" : {
            "value" : 10104.0,
            "normalized_value" : 336.8
          },
          "total" : {
            "value" : 20215.0,
            "normalized_value" : 673.8333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T07:42:30.000Z",
          "key" : 1606808550000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4463123.0
          },
          "primaries_raw" : {
            "value" : 2824172.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20340.0,
            "normalized_value" : 678.0
          }
        },
        {
          "key_as_string" : "2020-12-01T07:43:00.000Z",
          "key" : 1606808580000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 4483341.0
          },
          "primaries_raw" : {
            "value" : 2834281.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:43:30.000Z",
          "key" : 1606808610000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 4503681.0
          },
          "primaries_raw" : {
            "value" : 2844451.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20340.0,
            "normalized_value" : 678.0
          }
        },
        {
          "key_as_string" : "2020-12-01T07:44:00.000Z",
          "key" : 1606808640000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4523899.0
          },
          "primaries_raw" : {
            "value" : 2854560.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:44:30.000Z",
          "key" : 1606808670000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4544239.0
          },
          "primaries_raw" : {
            "value" : 2864730.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20340.0,
            "normalized_value" : 678.0
          }
        },
        {
          "key_as_string" : "2020-12-01T07:45:00.000Z",
          "key" : 1606808700000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4564457.0
          },
          "primaries_raw" : {
            "value" : 2874839.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:45:30.000Z",
          "key" : 1606808730000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4584799.0
          },
          "primaries_raw" : {
            "value" : 2885011.0
          },
          "primaries" : {
            "value" : 10172.0,
            "normalized_value" : 339.06666666666666
          },
          "total" : {
            "value" : 20342.0,
            "normalized_value" : 678.0666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:46:00.000Z",
          "key" : 1606808760000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4605015.0
          },
          "primaries_raw" : {
            "value" : 2895118.0
          },
          "primaries" : {
            "value" : 10107.0,
            "normalized_value" : 336.9
          },
          "total" : {
            "value" : 20216.0,
            "normalized_value" : 673.8666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:46:30.000Z",
          "key" : 1606808790000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4625364.0
          },
          "primaries_raw" : {
            "value" : 2905296.0
          },
          "primaries" : {
            "value" : 10178.0,
            "normalized_value" : 339.26666666666665
          },
          "total" : {
            "value" : 20349.0,
            "normalized_value" : 678.3
          }
        },
        {
          "key_as_string" : "2020-12-01T07:47:00.000Z",
          "key" : 1606808820000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 4645573.0
          },
          "primaries_raw" : {
            "value" : 2915397.0
          },
          "primaries" : {
            "value" : 10101.0,
            "normalized_value" : 336.7
          },
          "total" : {
            "value" : 20209.0,
            "normalized_value" : 673.6333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:47:30.000Z",
          "key" : 1606808850000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 4665929.0
          },
          "primaries_raw" : {
            "value" : 2925567.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20356.0,
            "normalized_value" : 678.5333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:48:00.000Z",
          "key" : 1606808880000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4686139.0
          },
          "primaries_raw" : {
            "value" : 2935676.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20210.0,
            "normalized_value" : 673.6666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T07:48:30.000Z",
          "key" : 1606808910000,
          "doc_count" : 6639,
          "total_raw" : {
            "value" : 4706495.0
          },
          "primaries_raw" : {
            "value" : 2945846.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20356.0,
            "normalized_value" : 678.5333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:49:00.000Z",
          "key" : 1606808940000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4720174.0
          },
          "primaries_raw" : {
            "value" : 2952698.0
          },
          "primaries" : {
            "value" : 6852.0,
            "normalized_value" : 228.4
          },
          "total" : {
            "value" : 13679.0,
            "normalized_value" : 455.96666666666664
          }
        },
        {
          "key_as_string" : "2020-12-01T07:49:30.000Z",
          "key" : 1606808970000,
          "doc_count" : 6639,
          "total_raw" : {
            "value" : 4740513.0
          },
          "primaries_raw" : {
            "value" : 2962867.0
          },
          "primaries" : {
            "value" : 10169.0,
            "normalized_value" : 338.96666666666664
          },
          "total" : {
            "value" : 20339.0,
            "normalized_value" : 677.9666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:50:00.000Z",
          "key" : 1606809000000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4754215.0
          },
          "primaries_raw" : {
            "value" : 2969718.0
          },
          "primaries" : {
            "value" : 6851.0,
            "normalized_value" : 228.36666666666667
          },
          "total" : {
            "value" : 13702.0,
            "normalized_value" : 456.73333333333335
          }
        },
        {
          "key_as_string" : "2020-12-01T07:50:30.000Z",
          "key" : 1606809030000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 4774571.0
          },
          "primaries_raw" : {
            "value" : 2979888.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20356.0,
            "normalized_value" : 678.5333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:51:00.000Z",
          "key" : 1606809060000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 4794774.0
          },
          "primaries_raw" : {
            "value" : 2989998.0
          },
          "primaries" : {
            "value" : 10110.0,
            "normalized_value" : 337.0
          },
          "total" : {
            "value" : 20203.0,
            "normalized_value" : 673.4333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:51:30.000Z",
          "key" : 1606809090000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4815131.0
          },
          "primaries_raw" : {
            "value" : 3000168.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20357.0,
            "normalized_value" : 678.5666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:52:00.000Z",
          "key" : 1606809120000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4835333.0
          },
          "primaries_raw" : {
            "value" : 3010277.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20202.0,
            "normalized_value" : 673.4
          }
        },
        {
          "key_as_string" : "2020-12-01T07:52:30.000Z",
          "key" : 1606809150000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4855691.0
          },
          "primaries_raw" : {
            "value" : 3020448.0
          },
          "primaries" : {
            "value" : 10171.0,
            "normalized_value" : 339.03333333333336
          },
          "total" : {
            "value" : 20358.0,
            "normalized_value" : 678.6
          }
        },
        {
          "key_as_string" : "2020-12-01T07:53:00.000Z",
          "key" : 1606809180000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 4875893.0
          },
          "primaries_raw" : {
            "value" : 3030557.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20202.0,
            "normalized_value" : 673.4
          }
        },
        {
          "key_as_string" : "2020-12-01T07:53:30.000Z",
          "key" : 1606809210000,
          "doc_count" : 6640,
          "total_raw" : {
            "value" : 4889749.0
          },
          "primaries_raw" : {
            "value" : 3037477.0
          },
          "primaries" : {
            "value" : 6920.0,
            "normalized_value" : 230.66666666666666
          },
          "total" : {
            "value" : 13856.0,
            "normalized_value" : 461.8666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:54:00.000Z",
          "key" : 1606809240000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4909951.0
          },
          "primaries_raw" : {
            "value" : 3047578.0
          },
          "primaries" : {
            "value" : 10101.0,
            "normalized_value" : 336.7
          },
          "total" : {
            "value" : 20202.0,
            "normalized_value" : 673.4
          }
        },
        {
          "key_as_string" : "2020-12-01T07:54:30.000Z",
          "key" : 1606809270000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4930291.0
          },
          "primaries_raw" : {
            "value" : 3057748.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20340.0,
            "normalized_value" : 678.0
          }
        },
        {
          "key_as_string" : "2020-12-01T07:55:00.000Z",
          "key" : 1606809300000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4950493.0
          },
          "primaries_raw" : {
            "value" : 3067857.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20202.0,
            "normalized_value" : 673.4
          }
        },
        {
          "key_as_string" : "2020-12-01T07:55:30.000Z",
          "key" : 1606809330000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4970835.0
          },
          "primaries_raw" : {
            "value" : 3078028.0
          },
          "primaries" : {
            "value" : 10171.0,
            "normalized_value" : 339.03333333333336
          },
          "total" : {
            "value" : 20342.0,
            "normalized_value" : 678.0666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:56:00.000Z",
          "key" : 1606809360000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 4991060.0
          },
          "primaries_raw" : {
            "value" : 3088144.0
          },
          "primaries" : {
            "value" : 10116.0,
            "normalized_value" : 337.2
          },
          "total" : {
            "value" : 20225.0,
            "normalized_value" : 674.1666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T07:56:30.000Z",
          "key" : 1606809390000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5011395.0
          },
          "primaries_raw" : {
            "value" : 3098308.0
          },
          "primaries" : {
            "value" : 10164.0,
            "normalized_value" : 338.8
          },
          "total" : {
            "value" : 20335.0,
            "normalized_value" : 677.8333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T07:57:00.000Z",
          "key" : 1606809420000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 5031617.0
          },
          "primaries_raw" : {
            "value" : 3108419.0
          },
          "primaries" : {
            "value" : 10111.0,
            "normalized_value" : 337.03333333333336
          },
          "total" : {
            "value" : 20222.0,
            "normalized_value" : 674.0666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T07:57:30.000Z",
          "key" : 1606809450000,
          "doc_count" : 6639,
          "total_raw" : {
            "value" : 5050827.0
          },
          "primaries_raw" : {
            "value" : 3118028.0
          },
          "primaries" : {
            "value" : 9609.0,
            "normalized_value" : 320.3
          },
          "total" : {
            "value" : 19210.0,
            "normalized_value" : 640.3333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T07:58:00.000Z",
          "key" : 1606809480000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5065659.0
          },
          "primaries_raw" : {
            "value" : 3125440.0
          },
          "primaries" : {
            "value" : 7412.0,
            "normalized_value" : 247.06666666666666
          },
          "total" : {
            "value" : 14832.0,
            "normalized_value" : 494.4
          }
        },
        {
          "key_as_string" : "2020-12-01T07:58:30.000Z",
          "key" : 1606809510000,
          "doc_count" : 6639,
          "total_raw" : {
            "value" : 5079483.0
          },
          "primaries_raw" : {
            "value" : 3132352.0
          },
          "primaries" : {
            "value" : 6912.0,
            "normalized_value" : 230.4
          },
          "total" : {
            "value" : 13824.0,
            "normalized_value" : 460.8
          }
        },
        {
          "key_as_string" : "2020-12-01T07:59:00.000Z",
          "key" : 1606809540000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5099701.0
          },
          "primaries_raw" : {
            "value" : 3142461.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T07:59:30.000Z",
          "key" : 1606809570000,
          "doc_count" : 6640,
          "total_raw" : {
            "value" : 5113469.0
          },
          "primaries_raw" : {
            "value" : 3149345.0
          },
          "primaries" : {
            "value" : 6884.0,
            "normalized_value" : 229.46666666666667
          },
          "total" : {
            "value" : 13768.0,
            "normalized_value" : 458.93333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:00:00.000Z",
          "key" : 1606809600000,
          "doc_count" : 6639,
          "total_raw" : {
            "value" : 5127177.0
          },
          "primaries_raw" : {
            "value" : 3156202.0
          },
          "primaries" : {
            "value" : 6857.0,
            "normalized_value" : 228.56666666666666
          },
          "total" : {
            "value" : 13708.0,
            "normalized_value" : 456.93333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:00:30.000Z",
          "key" : 1606809630000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5147573.0
          },
          "primaries_raw" : {
            "value" : 3166400.0
          },
          "primaries" : {
            "value" : 10198.0,
            "normalized_value" : 339.93333333333334
          },
          "total" : {
            "value" : 20396.0,
            "normalized_value" : 679.8666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:01:00.000Z",
          "key" : 1606809660000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5167785.0
          },
          "primaries_raw" : {
            "value" : 3176503.0
          },
          "primaries" : {
            "value" : 10103.0,
            "normalized_value" : 336.76666666666665
          },
          "total" : {
            "value" : 20212.0,
            "normalized_value" : 673.7333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:01:30.000Z",
          "key" : 1606809690000,
          "doc_count" : 6638,
          "total_raw" : {
            "value" : 5181618.0
          },
          "primaries_raw" : {
            "value" : 3183423.0
          },
          "primaries" : {
            "value" : 6920.0,
            "normalized_value" : 230.66666666666666
          },
          "total" : {
            "value" : 13833.0,
            "normalized_value" : 461.1
          }
        },
        {
          "key_as_string" : "2020-12-01T08:02:00.000Z",
          "key" : 1606809720000,
          "doc_count" : 6640,
          "total_raw" : {
            "value" : 5201827.0
          },
          "primaries_raw" : {
            "value" : 3193524.0
          },
          "primaries" : {
            "value" : 10101.0,
            "normalized_value" : 336.7
          },
          "total" : {
            "value" : 20209.0,
            "normalized_value" : 673.6333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:02:30.000Z",
          "key" : 1606809750000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5215659.0
          },
          "primaries_raw" : {
            "value" : 3200444.0
          },
          "primaries" : {
            "value" : 6920.0,
            "normalized_value" : 230.66666666666666
          },
          "total" : {
            "value" : 13832.0,
            "normalized_value" : 461.06666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:03:00.000Z",
          "key" : 1606809780000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5235877.0
          },
          "primaries_raw" : {
            "value" : 3210553.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:03:30.000Z",
          "key" : 1606809810000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5256225.0
          },
          "primaries_raw" : {
            "value" : 3220723.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20348.0,
            "normalized_value" : 678.2666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:04:00.000Z",
          "key" : 1606809840000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5276433.0
          },
          "primaries_raw" : {
            "value" : 3230830.0
          },
          "primaries" : {
            "value" : 10107.0,
            "normalized_value" : 336.9
          },
          "total" : {
            "value" : 20208.0,
            "normalized_value" : 673.6
          }
        },
        {
          "key_as_string" : "2020-12-01T08:04:30.000Z",
          "key" : 1606809870000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 5296767.0
          },
          "primaries_raw" : {
            "value" : 3240994.0
          },
          "primaries" : {
            "value" : 10164.0,
            "normalized_value" : 338.8
          },
          "total" : {
            "value" : 20334.0,
            "normalized_value" : 677.8
          }
        },
        {
          "key_as_string" : "2020-12-01T08:05:00.000Z",
          "key" : 1606809900000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5316985.0
          },
          "primaries_raw" : {
            "value" : 3251103.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:05:30.000Z",
          "key" : 1606809930000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5337326.0
          },
          "primaries_raw" : {
            "value" : 3261274.0
          },
          "primaries" : {
            "value" : 10171.0,
            "normalized_value" : 339.03333333333336
          },
          "total" : {
            "value" : 20341.0,
            "normalized_value" : 678.0333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:06:00.000Z",
          "key" : 1606809960000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5357543.0
          },
          "primaries_raw" : {
            "value" : 3271382.0
          },
          "primaries" : {
            "value" : 10108.0,
            "normalized_value" : 336.93333333333334
          },
          "total" : {
            "value" : 20217.0,
            "normalized_value" : 673.9
          }
        },
        {
          "key_as_string" : "2020-12-01T08:06:30.000Z",
          "key" : 1606809990000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5372585.0
          },
          "primaries_raw" : {
            "value" : 3278903.0
          },
          "primaries" : {
            "value" : 7521.0,
            "normalized_value" : 250.7
          },
          "total" : {
            "value" : 15042.0,
            "normalized_value" : 501.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:07:00.000Z",
          "key" : 1606810020000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 5398105.0
          },
          "primaries_raw" : {
            "value" : 3291663.0
          },
          "primaries" : {
            "value" : 12760.0,
            "normalized_value" : 425.3333333333333
          },
          "total" : {
            "value" : 25520.0,
            "normalized_value" : 850.6666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:07:30.000Z",
          "key" : 1606810050000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5418463.0
          },
          "primaries_raw" : {
            "value" : 3301834.0
          },
          "primaries" : {
            "value" : 10171.0,
            "normalized_value" : 339.03333333333336
          },
          "total" : {
            "value" : 20358.0,
            "normalized_value" : 678.6
          }
        },
        {
          "key_as_string" : "2020-12-01T08:08:00.000Z",
          "key" : 1606810080000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5438679.0
          },
          "primaries_raw" : {
            "value" : 3311944.0
          },
          "primaries" : {
            "value" : 10110.0,
            "normalized_value" : 337.0
          },
          "total" : {
            "value" : 20216.0,
            "normalized_value" : 673.8666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:08:30.000Z",
          "key" : 1606810110000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 5459021.0
          },
          "primaries_raw" : {
            "value" : 3322113.0
          },
          "primaries" : {
            "value" : 10169.0,
            "normalized_value" : 338.96666666666664
          },
          "total" : {
            "value" : 20342.0,
            "normalized_value" : 678.0666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:09:00.000Z",
          "key" : 1606810140000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5479231.0
          },
          "primaries_raw" : {
            "value" : 3332222.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20210.0,
            "normalized_value" : 673.6666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:09:30.000Z",
          "key" : 1606810170000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5499563.0
          },
          "primaries_raw" : {
            "value" : 3342392.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20332.0,
            "normalized_value" : 677.7333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:10:00.000Z",
          "key" : 1606810200000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5519797.0
          },
          "primaries_raw" : {
            "value" : 3352509.0
          },
          "primaries" : {
            "value" : 10117.0,
            "normalized_value" : 337.23333333333335
          },
          "total" : {
            "value" : 20234.0,
            "normalized_value" : 674.4666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:10:30.000Z",
          "key" : 1606810230000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5540124.0
          },
          "primaries_raw" : {
            "value" : 3362673.0
          },
          "primaries" : {
            "value" : 10164.0,
            "normalized_value" : 338.8
          },
          "total" : {
            "value" : 20327.0,
            "normalized_value" : 677.5666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:11:00.000Z",
          "key" : 1606810260000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5560341.0
          },
          "primaries_raw" : {
            "value" : 3372781.0
          },
          "primaries" : {
            "value" : 10108.0,
            "normalized_value" : 336.93333333333334
          },
          "total" : {
            "value" : 20217.0,
            "normalized_value" : 673.9
          }
        },
        {
          "key_as_string" : "2020-12-01T08:11:30.000Z",
          "key" : 1606810290000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5580699.0
          },
          "primaries_raw" : {
            "value" : 3382952.0
          },
          "primaries" : {
            "value" : 10171.0,
            "normalized_value" : 339.03333333333336
          },
          "total" : {
            "value" : 20358.0,
            "normalized_value" : 678.6
          }
        },
        {
          "key_as_string" : "2020-12-01T08:12:00.000Z",
          "key" : 1606810320000,
          "doc_count" : 9898,
          "total_raw" : {
            "value" : 5600921.0
          },
          "primaries_raw" : {
            "value" : 3393063.0
          },
          "primaries" : {
            "value" : 10111.0,
            "normalized_value" : 337.03333333333336
          },
          "total" : {
            "value" : 20222.0,
            "normalized_value" : 674.0666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:12:30.000Z",
          "key" : 1606810350000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5621261.0
          },
          "primaries_raw" : {
            "value" : 3403233.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20340.0,
            "normalized_value" : 678.0
          }
        },
        {
          "key_as_string" : "2020-12-01T08:13:00.000Z",
          "key" : 1606810380000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5641463.0
          },
          "primaries_raw" : {
            "value" : 3413342.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20202.0,
            "normalized_value" : 673.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:13:30.000Z",
          "key" : 1606810410000,
          "doc_count" : 9897,
          "total_raw" : {
            "value" : 5661803.0
          },
          "primaries_raw" : {
            "value" : 3423512.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20340.0,
            "normalized_value" : 678.0
          }
        },
        {
          "key_as_string" : "2020-12-01T08:14:00.000Z",
          "key" : 1606810440000,
          "doc_count" : 9896,
          "total_raw" : {
            "value" : 5682021.0
          },
          "primaries_raw" : {
            "value" : 3433621.0
          },
          "primaries" : {
            "value" : 10109.0,
            "normalized_value" : 336.96666666666664
          },
          "total" : {
            "value" : 20218.0,
            "normalized_value" : 673.9333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:14:30.000Z",
          "key" : 1606810470000,
          "doc_count" : 6640,
          "total_raw" : {
            "value" : 5702369.0
          },
          "primaries_raw" : {
            "value" : 3443791.0
          },
          "primaries" : {
            "value" : 10170.0,
            "normalized_value" : 339.0
          },
          "total" : {
            "value" : 20348.0,
            "normalized_value" : 678.2666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:15:00.000Z",
          "key" : 1606810500000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5703047.0
          },
          "primaries_raw" : {
            "value" : 3444126.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 678.0,
            "normalized_value" : 22.6
          }
        },
        {
          "key_as_string" : "2020-12-01T08:15:30.000Z",
          "key" : 1606810530000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5703839.0
          },
          "primaries_raw" : {
            "value" : 3444522.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 792.0,
            "normalized_value" : 26.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:16:00.000Z",
          "key" : 1606810560000,
          "doc_count" : 3381,
          "total_raw" : {
            "value" : 5704494.0
          },
          "primaries_raw" : {
            "value" : 3444858.0
          },
          "primaries" : {
            "value" : 336.0,
            "normalized_value" : 11.2
          },
          "total" : {
            "value" : 655.0,
            "normalized_value" : 21.833333333333332
          }
        },
        {
          "key_as_string" : "2020-12-01T08:16:30.000Z",
          "key" : 1606810590000,
          "doc_count" : 3381,
          "total_raw" : {
            "value" : 5718325.0
          },
          "primaries_raw" : {
            "value" : 3451777.0
          },
          "primaries" : {
            "value" : 6919.0,
            "normalized_value" : 230.63333333333333
          },
          "total" : {
            "value" : 13831.0,
            "normalized_value" : 461.03333333333336
          }
        },
        {
          "key_as_string" : "2020-12-01T08:17:00.000Z",
          "key" : 1606810620000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5718987.0
          },
          "primaries_raw" : {
            "value" : 3452104.0
          },
          "primaries" : {
            "value" : 327.0,
            "normalized_value" : 10.9
          },
          "total" : {
            "value" : 662.0,
            "normalized_value" : 22.066666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:17:30.000Z",
          "key" : 1606810650000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5719788.0
          },
          "primaries_raw" : {
            "value" : 3452508.0
          },
          "primaries" : {
            "value" : 404.0,
            "normalized_value" : 13.466666666666667
          },
          "total" : {
            "value" : 801.0,
            "normalized_value" : 26.7
          }
        },
        {
          "key_as_string" : "2020-12-01T08:18:00.000Z",
          "key" : 1606810680000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5720457.0
          },
          "primaries_raw" : {
            "value" : 3452843.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 669.0,
            "normalized_value" : 22.3
          }
        },
        {
          "key_as_string" : "2020-12-01T08:18:30.000Z",
          "key" : 1606810710000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5721241.0
          },
          "primaries_raw" : {
            "value" : 3453231.0
          },
          "primaries" : {
            "value" : 388.0,
            "normalized_value" : 12.933333333333334
          },
          "total" : {
            "value" : 784.0,
            "normalized_value" : 26.133333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:19:00.000Z",
          "key" : 1606810740000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5721912.0
          },
          "primaries_raw" : {
            "value" : 3453567.0
          },
          "primaries" : {
            "value" : 336.0,
            "normalized_value" : 11.2
          },
          "total" : {
            "value" : 671.0,
            "normalized_value" : 22.366666666666667
          }
        },
        {
          "key_as_string" : "2020-12-01T08:19:30.000Z",
          "key" : 1606810770000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5722705.0
          },
          "primaries_raw" : {
            "value" : 3453967.0
          },
          "primaries" : {
            "value" : 400.0,
            "normalized_value" : 13.333333333333334
          },
          "total" : {
            "value" : 793.0,
            "normalized_value" : 26.433333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:20:00.000Z",
          "key" : 1606810800000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5723373.0
          },
          "primaries_raw" : {
            "value" : 3454297.0
          },
          "primaries" : {
            "value" : 330.0,
            "normalized_value" : 11.0
          },
          "total" : {
            "value" : 668.0,
            "normalized_value" : 22.266666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:20:30.000Z",
          "key" : 1606810830000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5724110.0
          },
          "primaries_raw" : {
            "value" : 3454666.0
          },
          "primaries" : {
            "value" : 369.0,
            "normalized_value" : 12.3
          },
          "total" : {
            "value" : 737.0,
            "normalized_value" : 24.566666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:21:00.000Z",
          "key" : 1606810860000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5724844.0
          },
          "primaries_raw" : {
            "value" : 3455036.0
          },
          "primaries" : {
            "value" : 370.0,
            "normalized_value" : 12.333333333333334
          },
          "total" : {
            "value" : 734.0,
            "normalized_value" : 24.466666666666665
          }
        },
        {
          "key_as_string" : "2020-12-01T08:21:30.000Z",
          "key" : 1606810890000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5725631.0
          },
          "primaries_raw" : {
            "value" : 3455426.0
          },
          "primaries" : {
            "value" : 390.0,
            "normalized_value" : 13.0
          },
          "total" : {
            "value" : 787.0,
            "normalized_value" : 26.233333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:22:00.000Z",
          "key" : 1606810920000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5726245.0
          },
          "primaries_raw" : {
            "value" : 3455733.0
          },
          "primaries" : {
            "value" : 307.0,
            "normalized_value" : 10.233333333333333
          },
          "total" : {
            "value" : 614.0,
            "normalized_value" : 20.466666666666665
          }
        },
        {
          "key_as_string" : "2020-12-01T08:22:30.000Z",
          "key" : 1606810950000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5727097.0
          },
          "primaries_raw" : {
            "value" : 3456163.0
          },
          "primaries" : {
            "value" : 430.0,
            "normalized_value" : 14.333333333333334
          },
          "total" : {
            "value" : 852.0,
            "normalized_value" : 28.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:23:00.000Z",
          "key" : 1606810980000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5727765.0
          },
          "primaries_raw" : {
            "value" : 3456493.0
          },
          "primaries" : {
            "value" : 330.0,
            "normalized_value" : 11.0
          },
          "total" : {
            "value" : 668.0,
            "normalized_value" : 22.266666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:23:30.000Z",
          "key" : 1606811010000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5728557.0
          },
          "primaries_raw" : {
            "value" : 3456889.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 792.0,
            "normalized_value" : 26.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:24:00.000Z",
          "key" : 1606811040000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5729236.0
          },
          "primaries_raw" : {
            "value" : 3457232.0
          },
          "primaries" : {
            "value" : 343.0,
            "normalized_value" : 11.433333333333334
          },
          "total" : {
            "value" : 679.0,
            "normalized_value" : 22.633333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:24:30.000Z",
          "key" : 1606811070000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5730020.0
          },
          "primaries_raw" : {
            "value" : 3457621.0
          },
          "primaries" : {
            "value" : 389.0,
            "normalized_value" : 12.966666666666667
          },
          "total" : {
            "value" : 784.0,
            "normalized_value" : 26.133333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:25:00.000Z",
          "key" : 1606811100000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5730689.0
          },
          "primaries_raw" : {
            "value" : 3457955.0
          },
          "primaries" : {
            "value" : 334.0,
            "normalized_value" : 11.133333333333333
          },
          "total" : {
            "value" : 669.0,
            "normalized_value" : 22.3
          }
        },
        {
          "key_as_string" : "2020-12-01T08:25:30.000Z",
          "key" : 1606811130000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5731490.0
          },
          "primaries_raw" : {
            "value" : 3458359.0
          },
          "primaries" : {
            "value" : 404.0,
            "normalized_value" : 13.466666666666667
          },
          "total" : {
            "value" : 801.0,
            "normalized_value" : 26.7
          }
        },
        {
          "key_as_string" : "2020-12-01T08:26:00.000Z",
          "key" : 1606811160000,
          "doc_count" : 3381,
          "total_raw" : {
            "value" : 5732154.0
          },
          "primaries_raw" : {
            "value" : 3458688.0
          },
          "primaries" : {
            "value" : 329.0,
            "normalized_value" : 10.966666666666667
          },
          "total" : {
            "value" : 664.0,
            "normalized_value" : 22.133333333333333
          }
        },
        {
          "key_as_string" : "2020-12-01T08:26:30.000Z",
          "key" : 1606811190000,
          "doc_count" : 3381,
          "total_raw" : {
            "value" : 5745987.0
          },
          "primaries_raw" : {
            "value" : 3465608.0
          },
          "primaries" : {
            "value" : 6920.0,
            "normalized_value" : 230.66666666666666
          },
          "total" : {
            "value" : 13833.0,
            "normalized_value" : 461.1
          }
        },
        {
          "key_as_string" : "2020-12-01T08:27:00.000Z",
          "key" : 1606811220000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5746653.0
          },
          "primaries_raw" : {
            "value" : 3465937.0
          },
          "primaries" : {
            "value" : 329.0,
            "normalized_value" : 10.966666666666667
          },
          "total" : {
            "value" : 666.0,
            "normalized_value" : 22.2
          }
        },
        {
          "key_as_string" : "2020-12-01T08:27:30.000Z",
          "key" : 1606811250000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5747453.0
          },
          "primaries_raw" : {
            "value" : 3466333.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 800.0,
            "normalized_value" : 26.666666666666668
          }
        },
        {
          "key_as_string" : "2020-12-01T08:28:00.000Z",
          "key" : 1606811280000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5748115.0
          },
          "primaries_raw" : {
            "value" : 3466668.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 662.0,
            "normalized_value" : 22.066666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:28:30.000Z",
          "key" : 1606811310000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5748923.0
          },
          "primaries_raw" : {
            "value" : 3467064.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 808.0,
            "normalized_value" : 26.933333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:29:00.000Z",
          "key" : 1606811340000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5749593.0
          },
          "primaries_raw" : {
            "value" : 3467399.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 670.0,
            "normalized_value" : 22.333333333333332
          }
        },
        {
          "key_as_string" : "2020-12-01T08:29:30.000Z",
          "key" : 1606811370000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5750367.0
          },
          "primaries_raw" : {
            "value" : 3467794.0
          },
          "primaries" : {
            "value" : 395.0,
            "normalized_value" : 13.166666666666666
          },
          "total" : {
            "value" : 774.0,
            "normalized_value" : 25.8
          }
        },
        {
          "key_as_string" : "2020-12-01T08:30:00.000Z",
          "key" : 1606811400000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5750979.0
          },
          "primaries_raw" : {
            "value" : 3468100.0
          },
          "primaries" : {
            "value" : 306.0,
            "normalized_value" : 10.2
          },
          "total" : {
            "value" : 612.0,
            "normalized_value" : 20.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:30:30.000Z",
          "key" : 1606811430000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5751843.0
          },
          "primaries_raw" : {
            "value" : 3468524.0
          },
          "primaries" : {
            "value" : 424.0,
            "normalized_value" : 14.133333333333333
          },
          "total" : {
            "value" : 864.0,
            "normalized_value" : 28.8
          }
        },
        {
          "key_as_string" : "2020-12-01T08:31:00.000Z",
          "key" : 1606811460000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5752497.0
          },
          "primaries_raw" : {
            "value" : 3468859.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 654.0,
            "normalized_value" : 21.8
          }
        },
        {
          "key_as_string" : "2020-12-01T08:31:30.000Z",
          "key" : 1606811490000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5753289.0
          },
          "primaries_raw" : {
            "value" : 3469255.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 792.0,
            "normalized_value" : 26.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:32:00.000Z",
          "key" : 1606811520000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5753959.0
          },
          "primaries_raw" : {
            "value" : 3469590.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 670.0,
            "normalized_value" : 22.333333333333332
          }
        },
        {
          "key_as_string" : "2020-12-01T08:32:30.000Z",
          "key" : 1606811550000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5754767.0
          },
          "primaries_raw" : {
            "value" : 3469986.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 808.0,
            "normalized_value" : 26.933333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:33:00.000Z",
          "key" : 1606811580000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5755421.0
          },
          "primaries_raw" : {
            "value" : 3470321.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 654.0,
            "normalized_value" : 21.8
          }
        },
        {
          "key_as_string" : "2020-12-01T08:33:30.000Z",
          "key" : 1606811610000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5756231.0
          },
          "primaries_raw" : {
            "value" : 3470718.0
          },
          "primaries" : {
            "value" : 397.0,
            "normalized_value" : 13.233333333333333
          },
          "total" : {
            "value" : 810.0,
            "normalized_value" : 27.0
          }
        },
        {
          "key_as_string" : "2020-12-01T08:34:00.000Z",
          "key" : 1606811640000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5756899.0
          },
          "primaries_raw" : {
            "value" : 3471052.0
          },
          "primaries" : {
            "value" : 334.0,
            "normalized_value" : 11.133333333333333
          },
          "total" : {
            "value" : 668.0,
            "normalized_value" : 22.266666666666666
          }
        },
        {
          "key_as_string" : "2020-12-01T08:34:30.000Z",
          "key" : 1606811670000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5757619.0
          },
          "primaries_raw" : {
            "value" : 3471417.0
          },
          "primaries" : {
            "value" : 365.0,
            "normalized_value" : 12.166666666666666
          },
          "total" : {
            "value" : 720.0,
            "normalized_value" : 24.0
          }
        },
        {
          "key_as_string" : "2020-12-01T08:35:00.000Z",
          "key" : 1606811700000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5758345.0
          },
          "primaries_raw" : {
            "value" : 3471783.0
          },
          "primaries" : {
            "value" : 366.0,
            "normalized_value" : 12.2
          },
          "total" : {
            "value" : 726.0,
            "normalized_value" : 24.2
          }
        },
        {
          "key_as_string" : "2020-12-01T08:35:30.000Z",
          "key" : 1606811730000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5759087.0
          },
          "primaries_raw" : {
            "value" : 3472150.0
          },
          "primaries" : {
            "value" : 367.0,
            "normalized_value" : 12.233333333333333
          },
          "total" : {
            "value" : 742.0,
            "normalized_value" : 24.733333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:36:00.000Z",
          "key" : 1606811760000,
          "doc_count" : 3380,
          "total_raw" : {
            "value" : 5759807.0
          },
          "primaries_raw" : {
            "value" : 3472514.0
          },
          "primaries" : {
            "value" : 364.0,
            "normalized_value" : 12.133333333333333
          },
          "total" : {
            "value" : 720.0,
            "normalized_value" : 24.0
          }
        },
        {
          "key_as_string" : "2020-12-01T08:36:30.000Z",
          "key" : 1606811790000,
          "doc_count" : 3382,
          "total_raw" : {
            "value" : 5773635.0
          },
          "primaries_raw" : {
            "value" : 3479428.0
          },
          "primaries" : {
            "value" : 6914.0,
            "normalized_value" : 230.46666666666667
          },
          "total" : {
            "value" : 13828.0,
            "normalized_value" : 460.93333333333334
          }
        },
        {
          "key_as_string" : "2020-12-01T08:37:00.000Z",
          "key" : 1606811820000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5774249.0
          },
          "primaries_raw" : {
            "value" : 3479735.0
          },
          "primaries" : {
            "value" : 307.0,
            "normalized_value" : 10.233333333333333
          },
          "total" : {
            "value" : 614.0,
            "normalized_value" : 20.466666666666665
          }
        },
        {
          "key_as_string" : "2020-12-01T08:37:30.000Z",
          "key" : 1606811850000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5775099.0
          },
          "primaries_raw" : {
            "value" : 3480160.0
          },
          "primaries" : {
            "value" : 425.0,
            "normalized_value" : 14.166666666666666
          },
          "total" : {
            "value" : 850.0,
            "normalized_value" : 28.333333333333332
          }
        },
        {
          "key_as_string" : "2020-12-01T08:38:00.000Z",
          "key" : 1606811880000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5775769.0
          },
          "primaries_raw" : {
            "value" : 3480495.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 670.0,
            "normalized_value" : 22.333333333333332
          }
        },
        {
          "key_as_string" : "2020-12-01T08:38:30.000Z",
          "key" : 1606811910000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5776561.0
          },
          "primaries_raw" : {
            "value" : 3480891.0
          },
          "primaries" : {
            "value" : 396.0,
            "normalized_value" : 13.2
          },
          "total" : {
            "value" : 792.0,
            "normalized_value" : 26.4
          }
        },
        {
          "key_as_string" : "2020-12-01T08:39:00.000Z",
          "key" : 1606811940000,
          "doc_count" : 123,
          "total_raw" : {
            "value" : 5777231.0
          },
          "primaries_raw" : {
            "value" : 3481226.0
          },
          "primaries" : {
            "value" : 335.0,
            "normalized_value" : 11.166666666666666
          },
          "total" : {
            "value" : 670.0,
            "normalized_value" : 22.333333333333332
          }
        },
        {
          "key_as_string" : "2020-12-01T08:39:30.000Z",
          "key" : 1606811970000,
          "doc_count" : 122,
          "total_raw" : {
            "value" : 5778032.0
          },
          "primaries_raw" : {
            "value" : 3481630.0
          },
          "primaries" : {
            "value" : 404.0,
            "normalized_value" : 13.466666666666667
          },
          "total" : {
            "value" : 801.0,
            "normalized_value" : 26.7
          }
        },
        {
          "key_as_string" : "2020-12-01T08:40:00.000Z",
          "key" : 1606812000000,
          "doc_count" : 124,
          "total_raw" : {
            "value" : 5778693.0
          },
          "primaries_raw" : {
            "value" : 3481957.0
          },
          "primaries" : {
            "value" : 327.0,
            "normalized_value" : 10.9
          },
          "total" : {
            "value" : 661.0,
            "normalized_value" : 22.033333333333335
          }
        },
        {
          "key_as_string" : "2020-12-01T08:40:30.000Z",
          "key" : 1606812030000,
          "doc_count" : 40,
          "total_raw" : {
            "value" : 5778912.0
          },
          "primaries_raw" : {
            "value" : 3482066.0
          },
          "primaries" : {
            "value" : 109.0,
            "normalized_value" : 3.6333333333333333
          },
          "total" : {
            "value" : 219.0,
            "normalized_value" : 7.3
          }
        }
      ]
    }
  }
}

image

fredbcode commented 3 years ago

Sorry better like this

image

chrisronline commented 3 years ago

Thanks @fredbcode

I just want to confirm - there is an hour time discrepancy between the results from the query and the chart from the screenshot you pasted. Does this data line up to the chart in the screenshot?

fredbcode commented 3 years ago

The chart from Kibana is always the same, very regular, low with a little up every five minutes and from api the result is also more or less the same from my query posted today

chrisronline commented 3 years ago

Let's look at a sample spike then.

{
  "key_as_string" : "2020-12-01T08:16:00.000Z",
  "key" : 1606810560000,
  "doc_count" : 3381,
  "total_raw" : {
    "value" : 5704494.0
  },
  "primaries_raw" : {
    "value" : 3444858.0
  },
  "primaries" : {
    "value" : 336.0,
    "normalized_value" : 11.2
  },
  "total" : {
    "value" : 655.0,
    "normalized_value" : 21.833333333333332
  }
},
{
  "key_as_string" : "2020-12-01T08:16:30.000Z",
  "key" : 1606810590000,
  "doc_count" : 3381,
  "total_raw" : {
    "value" : 5718325.0
  },
  "primaries_raw" : {
    "value" : 3451777.0
  },
  "primaries" : {
    "value" : 6919.0,
    "normalized_value" : 230.63333333333333
  },
  "total" : {
    "value" : 13831.0,
    "normalized_value" : 461.03333333333336
  }
},
{
  "key_as_string" : "2020-12-01T08:17:00.000Z",
  "key" : 1606810620000,
  "doc_count" : 123,
  "total_raw" : {
    "value" : 5718987.0
  },
  "primaries_raw" : {
    "value" : 3452104.0
  },
  "primaries" : {
    "value" : 327.0,
    "normalized_value" : 10.9
  },
  "total" : {
    "value" : 662.0,
    "normalized_value" : 22.066666666666666
  }
},
Time Total normalized Primaries normalized
08:16:00 21.833333333333332 11.2
08:16:30 461.03333333333336 230.63333333333333
08:17:00 22.066666666666666 10.9

This is what we're charting - the huge increase from 21 -> 461 which is why you see a blip in the chart.

fredbcode commented 3 years ago

Seems I misinterpreted this

image

chrisronline commented 3 years ago

What is that from? What is the configuration for that chart?

fredbcode commented 3 years ago

From just one of my index, same cluster of course, netflow indexes from logstash