Should we ignore empty values on keyword fields

[jimczi]

Today when an empty value is passed to a keyword field we store an empty value in the doc_values but not in the inverted lists. To quote @rjernst here:

this seems inconsistent because the doc values should be the inverse of indexed values

It would be weird to index an empty value since there is nothing to normalize/analyze in this case but we could ignore empty value entirely instead.

[elasticmachine]

Pinging @elastic/es-search-aggs

[polyfractal]

Chatted about this internally. The first action item is to document the current behavior so that users are aware of the behavior, regardless of what we decide.

The general sentiment is that we should be consistent here and not store empty strings in DV either. There were some concerns about conflating null/empty since other systems treat them separately. It was unclear how much users rely on empty strings, rather than just tolerating their existence because that's how DV (and other systems) work today.

If we decide to make DV consistent with the inverted index, there are several moving parts that need addressing:

• Exists query falls back to querying the _field_names field if both DV and norms are disabled. We would need to prevent the field from being added to _field_names (or filter it out of the exists search or something) to keep the behavior consistent
• missing option of the Terms aggregation relies on DV, meaning empty strings show up as distinct buckets. This would change and needs to be documented as a break
• Related, returning null from a scripted Terms agg currently NPEs, but should instead be accepted and treated as a missing value.
  • Interaction between null_value and nulls returned from a script. Should it get the null_value as well?
• How to do all this in a BWC manner

[droberts195]

/cc @elastic/ml-core because whatever is decided here probably affects ML more than most.

For anomaly detection #60141 is related. We actually have extra complexity within the ML C++ code to carefully treat not present and empty string the same way. We were thinking of removing this because it dates back to when we received data in CSV format, and CSV cannot distinguish not present and empty string (for a field that exists in at least one record in the file). JSON can distinguish not present from empty string, but this issue is proposing that Elasticsearch then mask that difference. There will still be an inconsistency though if people decide to look at _source.

Unlike anomaly detection data frame analytics is more recent functionality that has no special workarounds to make not present and empty string the same. Possibly we might need to change something there if this proposal is implemented.

Obviously the situations where these inconsistencies occur are edge cases and won't affect that many users. But when they do occur we need to make sure they are not causing results that lead to hard-to-diagnose support cases.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)