[Aggs] Apply hard_bounds to prevent ES circuit breaker when dealing with open-ended ranges

[wylieconlon]

For date histogram aggregations, we need to set the hard_bounds parameter when dealing with any date_range data type. Unlike extended_bounds, hard_bounds is exclusive, which means that to get the same results we will need to set hard_bounds to be larger than the possible time range by up to one bucket length on either side.

For example, if the date range is "now-7d" to "now", with an interval of 3 hours, then we need to set hard_bounds to now-7d-3h to now+3h to be accurate.

For histogram aggregations, we can't set hard_bounds automatically. This means that open-ended ranges in histogram aggregations can easily cause circuit breakers, so we should require users to provide a hard_bounds setting in this case.

This issue was originally found here: https://github.com/elastic/kibana/pull/76971

cc @ppisljar

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[ppisljar]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.

[simianhacker]

I'm re-opening this because there is currently a bug with the Lens visualization when visualizing open ended date_range fields. We (Actionable Observability) ran into this issue with: https://github.com/elastic/kibana/pull/141309

Here is an example you can run in the Dev Console:

PUT test_range
{
  "settings": {
    "number_of_shards": 2
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "id": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "status": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "time_frame": {
        "type": "date_range", 
        "format": "epoch_millis||strict_date_optional_time"
      }
    }
  }
}

PUT test_range/_doc/1
{
  "@timestamp": "2022-09-21T13:00:00Z",
  "alert_id": "1",
  "status": "active",
  "time_frame": {
    "gte": "2022-09-21T02:00:00Z"
  }
}

PUT test_range/_doc/2
{
  "@timestamp": "2022-09-21T13:00:00Z",
  "alert_id": "2",
  "status": "recovered",
  "time_frame": {
    "gte": "2022-09-21T02:00:00Z",
    "lte": "2022-09-21T13:00:00Z"
  }
}

POST test_range/_search
{
  "size": 1,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "time_frame": {
              "gte": "2022-09-21T00:00:00Z",
              "lte": "2022-09-21T23:59:59Z"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "timeseries": {
      "date_histogram": {
        "field": "time_frame",
        "fixed_interval": "1h",
        "extended_bounds": {
          "min": "2022-09-21T00:00:00Z",
          "max": "2022-09-21T23:59:59Z"
        },
        "hard_bounds": {
          "min": "2022-09-21T00:00:00Z",
          "max": "2022-09-21T23:59:59Z"
        }
      }
    }
  }
}

Both extended_bounds and hard_bounds need to be set to draw the empty buckets AND to limit the date_histogram from trying to draw the buckets (to infinity) for the open ended date_range events.

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[davismcphee]

Hi @ppisljar, can you explain why this issue was moved from Visualizations to Data Discovery? It's not clear to me.

[ppisljar]

I think AggConfigs (which this issue will affect) are owned by data-discovery ?

[stratoula]

We own aggs (as we are the only consimers) and data discovery owns search, I will transfer again the ownership. It is a bit vague which teams owns what, I totally get the confusion. :)