[Discover] [Unified Field List] Fields with dots in their name and `subobjects: false` in their mapping incorrectly appear as multi fields in the field list

[davismcphee]

When viewing data in Discover from an index that contains subobjects: false in its mappings, fields containing dots in their name incorrectly appear as multi fields in the field list. These fields are not actually multi fields and are independent of the field Discover identifies as their parent:

The actual mappings:

{
  "mappings": {
    "properties": {
      "metrics": {
        "subobjects": false,
        "properties": {
          "time": {
            "type": "long"
          },
          "time.max": {
            "type": "long"
          },
          "time.min": {
            "type": "long"
          }
        }
      }
    }
  }
}

Example data to reproduce:

PUT test-subobject
{
  "mappings": {
    "properties": {
      "metrics": {
        "type":  "object",
        "subobjects": false, 
        "properties": {
          "time": { "type": "long" },
          "time.min": { "type": "long" },
          "time.max": { "type": "long" }
        }
      }
    }
  }
}

PUT test-subobject/_doc/1
{
  "metrics.time" : 100, 
  "metrics.time.min" : 10,
  "metrics.time.max" : 900
}

POST kbn:/api/data_views/data_view
{
  "data_view": {
     "title": "test-subobject*",
     "name": "My test subobject Data View"
  }
}

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[mattkime]

I'm pretty sure this happens because multifields are inferred based on field_caps data. Generally speaking, we haven't been strict when it comes to field names and object hierarchy. ES isn't strict because it will ingest whatever its fed. There are expressive benefits to not distinguishing between "true" objects and field names containing dots. Obviously this creates problems when we need to infer structure based on a limited set of attributes.

https://github.com/elastic/kibana/blob/main/src/plugins/data_views/server/fetcher/lib/field_capabilities/field_caps_response.ts#L195

[kertal]

@davismcphee @mattkime I don't think we get this information by the field_caps request, right? In this case this is blocked, since we would need more information by the field_caps request, else we can't improve this behavior

[davismcphee]

@kertal Yes that sounds correct based on @mattkime's comment. I wasn't aware of that when I made the issue, but I'm not sure we can address this without upstream ES changes. Maybe @mattkime can confirm this is the case, and if so we'll add a blocked label.

[mattkime]

@kertal Yes, thats correct, its blocked.

[kertal]

thx, do we have an issue on ES side aiming to enrich field_caps to provide more info?