[Index pattern] Add support for object runtime field "subFields"

sebelga commented 3 years ago

:warning: [EDIT]: I've created https://github.com/elastic/kibana/discussions/99462 to discuss this proposal. Let's use that discussion to raise concerns and suggest alternatives/modifications. Once we have committed on the changed required I will update the description of the issue.

Problem

Object runtime fields give the user the possibility to declare multiple sub fields inside a parent object through a script. For that the user will emit multiple value from the painless script as a set of key/value pairs. The problem is: this single runtime field, once saved in the Kibana index pattern, will appear as one field and none of the sub fields will be exposed. When an app (Lens, Discover) will ask for the list of fields on the index pattern it will then receive this top object runtime field and none of the associated sub fields.

Proposal

I suggest the following:

New `subFields` prop

The field editor returns a new optional subFields property on the runtime field (following the naming convention in place for concrete fields)

{
  "name": "myObject",
  "type": "object",
  "script": "// script to emit multiple sub fields",
  "format": // Will never be there for object runtime fields
  "subFields": {
    "foo": {
      "format": { // format definition },
      "priority: 1
    },
    "bar": {
      "format": { ... },
      "priority": 2,
    }
  }
}

Add a `parent` (optional) parameter to index pattern field definition

this.fields.add({
  name,
  runtimeField,
  type: castEsToKbnFieldTypeName(runtimeField.type),
  aggregatable: true,
  searchable: true,
  count: 0,
  readFromDocValues: false,
  parent: 'myRuntimeFieldObject' // New
});

This will allow us the normal CRUD operation like other runtime fields:

List: We will return only the leaf object and not the parent object runtime field with the script (we follow the same behaviour that Kibana does today for object and nested fields). For that we need to:
- Update the code to add fields top an IndexPattern when loading the saved object to also add all the subFields of object runtime fields
- Filter out the runtime object field parent object when returning the list of fields. I am not sure where it is best to do that but will consult with @mattkime
Create: To create subFields we need to update addRuntimeField() to not only create the parent runtime field object but also all of its subFields
Edit: When editing an leaf object (e.g. myRuntimeObject.foo) we will:
- check that it is a runtime field
- if so: check if it has a parent which is also a runtime field (with a runtimeField definition + a script)
- if so: we would open the editor without the script section, just like any concrete field (with custom label, format, priority). We would add an info callout saying that this field comes from an object runtime field with a link to edit the parent object field.
Delete: To delete a leaf object field we would open a modal saying that this field has been generated by an object runtime field script and that the user needs to modify the parent's script. We would add a button for the user to open the field editor and edit the script in order to remove the child field.

Support format for sub fields

We also need to update the logic to add the format definition for each subField to the index pattern attributes (here: index pattern field editor)

cc @mattkime @stacey-gammon

elasticmachine commented 3 years ago

Pinging @elastic/kibana-app-services (Team:AppServices)

elasticmachine commented 3 years ago

Pinging @elastic/es-ui (Team:Elasticsearch UI)

timroes commented 3 years ago

All that sounds reasonable for me, I guess my only question would be: should the parent flag also be written for non runtime fields object fields, or will it be a specific thing for runtime fields?

sebelga commented 3 years ago

All that sounds reasonable for me, I guess my only question would be: should the parent flag also be written for non runtime fields object fields, or will it be a specific thing for runtime fields?

Any field could potentially define it but for the time being it would only be set when creating runtime fields.

sebelga commented 3 years ago

Fixed in https://github.com/elastic/kibana/pull/110226

elastic / kibana