Add circuit breaker to prevent OOM searching very large docs

[dimitris-athanasiou]

Description

Presume an index contains docs with a single text field. The values of that field can be arbitrarily huge docs. There can easily come a point where doing a simple search against that index and requesting an appropriate size of docs in a single response can OOM the node.

Could there be a circuit breaker that prevents the node from OOM?

Steps to reproduce:

• a dataset which the message field in some documents have the length larger than 700,000. (please contact @wwang500 for datasets)
• run below query:

  GET <test_dataset>/_search
  {
  "size": 1000,
  "query": { 
  "range": {
    "message_length": {
      "gte": 700000
    }
  }
  }
  }

Observed: Node will turn unhealthy and restart due to OOM.

[dimitris-athanasiou]

cc @romseygeek

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)