elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.65k stars 24.65k forks source link

More efficient implementation of ignore_above in synthetic source for flattened fields #110239

Open lkts opened 2 months ago

lkts commented 2 months ago

Description

110214 handles synthetic source for flattened fields when ignore_above is used (child fields ignored due to ignore_above are now present in synthetic source). In that PR if there is any ignored child fields encountered during parsing it stores the entire object in _ignored_source field. This can be optimized by storing only ignored child fields in dedicated stored fields and then merging such values with valid child fields read from doc values. This is similar to how ignore_malformed works with synthetic source for other field types. This optimization will reduce storage space needed to handle this case.

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-storage-engine (Team:StorageEngine)