Add user component templates on all automated template installations

[jbaiera]

During node start up we automatically install a number of index templates for a number of features (ILM, Logs, ML, etc). There is an expectation that users not modify these templates because their changes will be wiped out in the event that the templates are updated by the system. However, in some cases we would like to allow users to update the templates.

We propose that an empty component template be added for all composable templates we automatically install. This empty component template will be created if it does not exist, but will never be updated by the system. The priority for this "user" component template will be assigned so that it supersedes any components created by the system. This allows the user to customize this component without the system interfering when updating the original templates.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

@joshdover you might be interested in this issue, since you were working on the proposal for how to do user customizations for the Fleet templates

[matschaffer]

Would isolating mappings from settings like we do in metrics (https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/metrics-template.json#L8) at least allow someone to override the settings component template?

If an empty component template seemed strange, having settings templates present and available for override could be good I think.

[joshdover]

+1 on this idea. It'd also be great if the component templates for the built-in logs and metrics templates matched the naming convention we've been using for the templates installed for Elastic Agent integrations. I'd recommend (in this order, lowest to highest precedence):

• logs@settings
• logs@mappings
• data-streams@mappings
• logs@custom - this is the new template, never overwritten by the system

[jbaiera]

@joshdover would you be able to share where some templates are defined for the Elastic Agent integrations? I'd like to take a look at some existing examples in more detail. I'm mostly wondering how the logs@mappings and data-streams@mappings examples that you mention might differ from each other.

[jbaiera]

Would isolating mappings from settings like we do in metrics (https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/metrics-template.json#L8) at least allow someone to override the settings component template?

@matschaffer It looks like that settings component is also managed by the system, so while it could be useful in the mean time to update the resulting template without impacting mapping changes it still is subject to being overwritten every version if we move toward standardizing our versioning scheme for templates as per #84147

[matschaffer]

I guess my thinking was that if we had mappings and settings in isolated components, a user could provide a higher priority index template of their own that uses the system mapping plus a customized settings component.

Not sure what the "right" move here though is. Hopefully someone who's put more thought time into the index template system can comment (which might be you James :) )

[joshdover]

would you be able to share where some templates are defined for the Elastic Agent integrations? I'd like to take a look at some existing examples in more detail. I'm mostly wondering how the logs@mappings and data-streams@mappings examples that you mention might differ from each other.

There's not a static place to point you too since they're generated by application code, but you can easily see what they look like by spinning up an 8.2.0 snapshot on Cloud staging, and then visit /app/integrations/detail/system/settings in Kibana and clicking "Install". I do recommend using 8.2.0 and not a prior version because we did make some recent changes to how these templates are structured in https://github.com/elastic/kibana/pull/124013.

Also worth noting that integrations do not use the data-streams@mappings or (data-streams-mappings as it's currently named) component template, but it does appear that the default logs and metrics index templates do. It'd be helpful to understand the intention behind this template and whether or not we should be using them in the index templates for integrations as well. Here is the current content:

{
  "dynamic_templates": [
    {
      "match_ip": {
        "mapping": {
          "type": "ip"
        },
        "match_mapping_type": "string",
        "match": "ip"
      }
    },
    {
      "match_message": {
        "mapping": {
          "type": "match_only_text"
        },
        "match_mapping_type": "string",
        "match": "message"
      }
    },
    {
      "strings_as_keyword": {
        "mapping": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "match_mapping_type": "string"
      }
    }
  ],
  "date_detection": false,
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "ecs": {
      "properties": {
        "version": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "data_stream": {
      "properties": {
        "namespace": {
          "type": "constant_keyword"
        },
        "dataset": {
          "type": "constant_keyword"
        }
      }
    },
    "host": {
      "type": "object"
    }
  }
}