[Discover] Indicate unmapped fields per doc in the UI

[flash1293]

I just stumbled over a weird issue that took me a while to figure out.

Take this screenshot:

service.name looks like a totally normal field, but:

• I can't filter out the values (see the not-filter pill), but the first document still has a service.name value of filebeat which should be filtered out
• I don't get a summary on the field

It also happily tells me to filter on this field which won't work either:

What gives? The reason is that the service.name field is mapped somewhere in my index pattern, but not on one specific index which happens to have values for it.

To make this easier to understand, there should be a small indicator in the rendered cell if the value is not mapped and thus won't react to the query.

I think we are already fetching the information about what indices out of the whole index pattern a field is mapped on, we could use this information here to avoid this confusion.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

@flash1293 generally indicating the user that the field is not mapped in this case would make sense. Question is how to determine this, you're writing

I think we are already fetching the information about what indices out of the whole index pattern a field is mapped on, we could use this information here to avoid this confusion.

where are we doing this? we generally detect unmapped fields (this is not the case here, since it's just partly unmapped). We detect mapping conflicts (this is not the case here). field_caps gives us the information about all indices matching the request. what did I miss?

[flash1293]

I think we are filtering that part out on the Kibana server or so - field caps also gives info about what indices have different mappings.

Example:

PUT my-index-1
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "a": {
        "type": "keyword"
      },
      "b": {
        "type": "keyword"
      }
    }
  }
}

PUT my-index-2
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "a": {
        "type": "keyword"
      }
    }
  }
}

request field caps:

POST my-index-*/_field_caps?include_unmapped=true&filters=-metadata
{
  "fields": ["*"]
}

response:

{
  "indices": [
    "my-index-1",
    "my-index-2"
  ],
  "fields": {
    "a": {
      "keyword": {
        "type": "keyword",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true
      }
    },
    "b": {
      "keyword": {
        "type": "keyword",
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "my-index-1"
        ]
      },
      "unmapped": {
        "type": "unmapped",
        "metadata_field": false,
        "searchable": false,
        "aggregatable": false,
        "indices": [
          "my-index-2"
        ]
      }
    }
  }
}

[flash1293]

Field caps tells us b is unmapped in my-index-2, so if a row is rendered, we can check the _index of the doc and indicate that b is not mapped.

[kertal]

Oh, thx, so this is what I missed, adding include_unmapped=true should do the trick, didn't test but I think then this should server side be considered as a conflict? dear @mattkime , could you check?

[davismcphee]

We were considering using a similar approach during data source profile resolution for checking if specific fields belong to all indices covered by an index pattern, but decided against it due to potential performance / response size concerns with wide index patterns (covering potentially thousands of indices). Not necessarily a blocker, but definitely something to consider. Including this functionality as part of the data view fields API may mitigate the impact and allow us to take advantage of browser caching, but we'll want to evaluate the impact regardless.