Avoid OOM on poison documents

[markharwood]

In examining an OOM heap dump from a 7.12 user it was apparent that the culprit was a rogue document that introduced 360k(!) new fields and had a bunch of TextFieldMapper objects held in o.e.index.mapper.ParseContext.InternalParseContext.dynamicMappers that together consumed 1.5GB of RAM of 2GB heap.

I know we have circuit breakers for numbers of fields in mappers but perhaps these objects are allocated during parsing before we test that condition?

Either way seems like we need some added robustness here.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[williamrandolph]

The core-infra team discussed this in our sync today. We don't believe resiliency is the primary problem here.

If we have a configured limit on the number of fields in a mapping, the code for the dynamic mappers should use that limit to avoid allocating too many objects whether or not the memory usage threatens OOM. If the only way to catch this issue were by using a memory circuit breaker, we'd consider that a resilience issue, but there is likely a better way to catch this by improving dynamic mappers.

We think the search team is probably better equipped to handle this issue.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[ywelsch]

73713 covers the scenario when a document has many different field names. There are scenarios which the PR does not cover, e.g. if the document has an array with a crazy number of subobjects that all have the same field name. We currently add a dynamic mapping update for all these subobjects fields, and only merge those later into one. This would only be solved by interleaving dynamic mapping creation with parsing, which isn’t easy to do. We could somewhat cover this scenario by also interleaving the index.mapping.nested_objects.limit + some other checks with parsing.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)