Automatically load unmapped fields from _source?

[jpountz]

This relates to #80504 where we would like users to think about field values and make Elasticsearch responsible for figuring out the best way to retrieve them, e.g. doc_values when available with a fallback to _source otherwise. One challenge with this vision is text fields (#81246), another one is unmapped fields that this issue focuses on.

Almost all Elasticsearch APIs only consider fields that exist in the mappings. If a field isn't mapped, queries, aggregations, etc. will treat the field the same way as if it didn't exist. This is one of the reasons why scripting users have to worry about doc vs. _source today, unmapped fields are only available in _source.

One way to address this discrepancy would consist of introducing an abstraction layer in our scripting API so that if a field is unmapped, then the script will look up values from _source instead.

Another option would be to address this more broadly by improving mappings so that they would create transient runtime fields whenever a field that is used in a query, aggregation might exist in _source documents but isn't mapped. This would be quite similar to flattened fields where sub fields are not explicitly mapped, and Elasticsearch creates transient field entries at runtime that look and feel like regular keyword fields. This way, unmapped fields that exist in _source could not only be used in scripts, but also in any query or aggregation using the same semantics as fields of the keyword family (e.g. wildcard.)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jdconrad]

My preference would be the second solution presented here as that creates a better consistency between scripting and existing queries. We would need a way for users to opt-out/opt-in to avoid the trappy performance hit of extracting values from source. Either solution will require a change in the abstraction layer for how scripts access values in fields.

[javanna]

We have discussed this and, similarly to the recent discussion at https://github.com/elastic/elasticsearch/issues/80504#issuecomment-1165959827, we concluded that we would like to focus at first on how scripts access unmapped fields through the scripting fields API. We could lean on the existing runtime field implementations that load from _source. A question that we did not answer yet is around the expected data-type: source access in scripts is currently driven by the json type of the field that gets accessed, hence if we want feature parity with the existing mechanism loading everything as keyword would not be good enough. We will answer these questions once we have implemented loading from _source for all mapped field types.

Another option would be to address this more broadly by improving mappings so that they would create transient runtime fields whenever a field that is used in a query, aggregation might exist in _source documents but isn't mapped.

This reminds me of the dynamic:runtime behaviour. I do wonder if doing this automatically is required given that transient runtime fields can be easily added as part of the search request. Maybe this surfaces the need to express that everything that is not mapped or matching some pattern should load from _source (aka dynamic_templates as part of the search request)? Yet if we could load from_source automatically, why ask users to set it up? We will get back to this discussion at a later time.

[jpountz]

Another option would be to address this more broadly by improving mappings so that they would create transient runtime fields whenever a field that is used in a query, aggregation might exist in _source documents but isn't mapped.

Noting that @felixbarny opened https://github.com/elastic/elasticsearch/issues/88249 about this earlier today.

[jpountz]

I do wonder if doing this automatically is required given that transient runtime fields can be easily added as part of the search request.

I guess one argument in favor of doing it automatically is that users might not always know if the field exists in the mappings or not. For instance if you run _field_caps against two indices, where one has the field, and the other one has it in the _source but dynamic mappings are disabled, _field_caps will pretend that the field exists so the user wouldn't know that they need a request runtime field to be able to see the field in the second index.

[javanna]

Another problem I see with adding a transient runtime field as part of the search request is that it is applied globally and will override any indexed field with the same name, hence if some indices do have the field defined, they effectively lose access to it this way.

[romseygeek]

it is applied globally and will override any indexed field with the same name

Not necessarily? We could implement it at the MappingLookup level, so that if you ask for a non-existent field we create a runtime field for you. Then this would all happen at the shard level and wouldn't impact indexes that have a concrete field defined.

[javanna]

Not necessarily? We could implement it at the MappingLookup level

I guess that you are talking about what we could do, and the possibilities are almost endless :) but I am talking about how things currently work. Currently if you declare a runtime field in the search request it will override any field with same name in any index, right?

[felixbarny]

Here's a related discussion:

• https://github.com/elastic/elasticsearch/issues/86536

[javanna]

@felixbarny I was also thinking of #86536 , but I think, and please correct me if I am misunderstanding, that there is an important distinction between falling back to _source and falling back to runtime fields like proposed in #86536 : the latter needs a script to compute a value that was not present in previously indexed documents and can't lean on source fallback, while the former involves loading an existing value from _source as-is.

[felixbarny]

Yes, you're right.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)