TSDB: terms query with order by top_metrics on a counter field reports null values when applied on mixed index (non-TSDB rolled over to TSDB)

[dej611]

Elasticsearch Version

8.8.0-SNAPSHOT

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin -- 21.6.0 Darwin Kernel Version 21.6.0: ; root:xnu-8020.240.18.700.8~1/RELEASE_ARM64_T6000 arm64

Problem Description

This issue can be reproduced on a index which was originally non-TSDB then migrated to TSDB. This happens when using the k8s integration package in kibana.

When sending a terms agg using a top_metrics with a counter field to order results, depending on the selected time range results vary from correct to null:

• if the time range filter starts and end before the rollover timing then the correct data is reported
• if the time range filter starts before the rollover and ends after it, then null values are returned
• if the time range filter starts and ends after the rollover timing, then the correct data is reported.

No shards failures reported.

Steps to Reproduce

More details with query can be found in this Kibana issue: https://github.com/elastic/kibana/issues/157839#issuecomment-1549934027

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[martijnvg]

@dej611 Is it possible to add an ES reproduction? I'm wondering whether what documents exist in the two indices that match with the range filter. This could explain why the pie chart renders differently.

[dej611]

@tetianakravchenko created the environment where I managed to reproduce the issue.

[salvatore-campagna]

I tried to reproduce the issue unsuccessfully using the following YAML test which does the following:

• create a test1 index with mode: time_series
• create a test2 index (no time series)
• index data in both indices including the same fields and the same mappings and excluding time series settings for the non-time series index
• query the two indices using 3 queries with time ranges set as explained above (before rollover, cross rollover, after rollover)
• create a data stream
• add the two existing indices to the data stream
• query the two indices as before but using the data stream as the query target

I see the same results. For this reason I think the issue might be rollover-related.

---
"mixed time series mode":
  - skip:
      features: allowed_warnings
      version: " - 8.1.99"
      reason: tsdb indexing changed in 8.2.0

  - do:
      indices.create:
        index: test1
        body:
          settings:
            index:
              number_of_shards: 3
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
              dataset:
                type: keyword
              resource:
                type: keyword
              count:
                type: long

  - do:
      indices.create:
        index: test2
        body:
          settings:
            index:
              mode: time_series
              routing_path: [ metricset ]
              time_series:
                start_time: 2021-04-28T00:00:00Z
                end_time: 2021-04-29T00:00:00Z
              number_of_shards: 3
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
                time_series_dimension: true
              dataset:
                type: keyword
              resource:
                type: keyword
                time_series_dimension: true
              count:
                type: long
                time_series_metric: counter

  - do:
      bulk:
        refresh: true
        index: test1
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 5 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 6 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 10 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 11 }'

  - do:
      bulk:
        refresh: true
        index: test2
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 14 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 21 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 16 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 25 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 29 }'

  - do:
      search:
        index: test1,test2
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-27T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:53:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 13 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:55:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 11 }

  - do:
      search:
        index: test1,test2
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      search:
        index: test1,test2
        body:
          fields:
            - field: "@timestamp"
              format: date
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    format: strict_date_optional_time
                    gte: "2021-04-28T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
                shard_size: 25
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                          minimum_should_match: 1
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      allowed_warnings:
        - "index template [data-stream-template] has index patterns [data-*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [my-template] will take precedence during new index creation"
      indices.put_index_template:
        name: data-stream-template
        body:
          index_patterns: [ data-* ]
          data_stream: { }

  - do:
      indices.create_data_stream:
        name: data-stream-mixed
  - is_true: acknowledged

  - do:
      indices.modify_data_stream:
        body:
          actions:
            - add_backing_index:
                data_stream: "data-stream-mixed"
                index: "test1"
  - is_true: acknowledged

  - do:
      indices.modify_data_stream:
        body:
          actions:
            - add_backing_index:
                data_stream: "data-stream-mixed"
                index: "test2"
  - is_true: acknowledged

  - do:
      indices.get_data_stream:
        name: data-stream-mixed

  - length: { data_streams: 1 }
  - length: { data_streams.0.indices: 3 }
  - match: { data_streams.0.indices.0.index_name: "test2" }
  - match: { data_streams.0.indices.1.index_name: "test1" }

  - do:
      indices.get_mapping:
        index: "test2"

  - match: { test2.mappings.properties.@timestamp.type: "date" }
  - match: { test2.mappings.properties.count.type: "long" }
  - match: { test2.mappings.properties.count.time_series_metric: "counter" }

  - do:
      indices.get_mapping:
        index: "test1"

  - match: { test1.mappings.properties.@timestamp.type: "date" }
  - match: { test1.mappings.properties.count.type: "long" }
  - match: { test1.mappings.properties.count.time_series_metric: null }

  - do:
      search:
        index: data-stream-mixed
        body:
          size: 20
          query:
            match_all: { }

  - length: { hits.hits: 16 }

  - do:
      search:
        index: data-stream-mixed
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-27T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:53:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 13 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:55:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 11 }

  - do:
      search:
        index: data-stream-mixed
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      search:
        index: data-stream-mixed
        body:
          fields:
            - field: "@timestamp"
              format: date
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    format: strict_date_optional_time
                    gte: "2021-04-28T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
                shard_size: 25
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                          minimum_should_match: 1
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

[salvatore-campagna]

I did another test introducing a rollover operation for one of the non-time series indices but still I am not able to reproduce the issue.

---
"mixed time series mode":
  - skip:
      features: allowed_warnings
      version: " - 8.1.99"
      reason: tsdb indexing changed in 8.2.0

  - do:
      allowed_warnings:
        - "index template [blue] has index patterns [purple-index] matching patterns from existing older templates [global] with patterns (global => [*]); this template [blue] will take precedence during new index creation"
      indices.put_index_template:
        name: blue
        body:
          index_patterns: [ "test-00000*" ]
          template:
            mappings:
              properties:
                "@timestamp":
                  type: date
                metricset:
                  type: keyword
                dataset:
                  type: keyword
                resource:
                  type: keyword
                count:
                  type: long

  - do:
      indices.create:
        index: test-000001
        body:
          settings:
            index:
              number_of_shards: 1
              number_of_replicas: 1

  - do:
      indices.create:
        index: test-time-series-000001
        body:
          settings:
            index:
              mode: time_series
              routing_path: [ metricset ]
              time_series:
                start_time: 2021-04-28T00:00:00Z
                end_time: 2021-04-29T00:00:00Z
              number_of_shards: 1
              number_of_replicas: 1
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
                time_series_dimension: true
              dataset:
                type: keyword
              resource:
                type: keyword
                time_series_dimension: true
              count:
                type: long
                time_series_metric: counter

  - do:
      indices.put_alias:
        index: test-000001
        name: alias_test-000001

  - do:
      indices.put_alias:
        index: test-time-series-000001
        name: alias_test-time-series-000001

  - do:
      bulk:
        refresh: true
        index: test-000001
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 5 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 6 }'

  - do:
      indices.rollover:
        alias: "alias_test-000001"
        wait_for_active_shards: 1
  - match: { old_index: "test-000001" }
  - match: { new_index: "test-000002" }
  - match: { rolled_over: true }
  - match: { dry_run: false }

  - do:
      bulk:
        refresh: true
        index: test-000002
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 10 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 11 }'

  - do:
      bulk:
        refresh: true
        index: test-time-series-000001
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 14 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 21 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 16 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 25 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 29 }'

  - do:
      search:
        index: test-000001,test-000002,test-time-series-000001
        body:
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-27T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:53:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 13 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:55:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 11 }

  - do:
      search:
        index: test-000001,test-000002,test-time-series-000001
        body:
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      search:
        index: test-000001,test-000002,test-time-series-000001
        body:
          fields:
            - field: "@timestamp"
              format: date
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    format: strict_date_optional_time
                    gte: "2021-04-28T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
                shard_size: 25
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                          minimum_should_match: 1
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)