ILM: the ilm-history index template is not versioned

[andreidan]

Elasticsearch Version

8.0

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin

Problem Description

At the moment we use the IndexTemplateRegistry infrastructure to manage the index templates we want to ship with elasticsearch. The index templates however themselves can manage data streams. One example is the ilm-history index template managing the ilm-history-5 data stream.

If we'd attempt a change to the history index (say, we want to add a new field in the mapping) by bumping the template version, this will fail when running the new version because bumping the index template version changes the index_patterns the ilm-history template is matching, but the template is already used by the existing ilm-history-5 data stream.

The index template name should include the version in the name, so we create a new template when we make changes.

[NOTE] there might be other users of the IndexTemplateRegistry in this situation.

This would yield an exception like this at startup

[kanelbullar] error adding index template [ilm-history] for [index_lifecycle]
java.lang.IllegalArgumentException: composable template [ilm-history] with index patterns [ilm-history-6*], priority [2147483647] would cause data streams [ilm-history-5] to no longer match a data stream template
    at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateDataStreamsStillReferenced(MetadataIndexTemplateService.java:727) ~[elasticsearch-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
    at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateIndexTemplateV2(MetadataIndexTemplateService.java:662) ~[elasticsearch-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
    at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:614) ~[elasticsearch-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
    at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:497) ~[elasticsearch-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
    at org.elasticsearch.cluster.ClusterStateTaskExecutor$1.execute(ClusterStateTaskExecutor.java:142)

Steps to Reproduce

1. Make a change to the ilm-history index template and bump the template version (eg. https://github.com/andreidan/elasticsearch/commit/bd450cce744d2937e40c9913238ad791760a8a67)
2. Build a distribution with the change ( eg. ./gradlew clean :distribution:archives:darwin-tar:assemble )
3. Run a 7.17 distribution and run ILM to register some steps in the ilm history.
4. Upgrade an existing 7.17 to the distribution you built at 2.

Logs (if relevant)

No response

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[jakelandis]

It looks like ilm-history might be in good company. From a new 8.0 development cluster:

curl -s localhost:9200/_index_template | jq '.index_templates[] | .name'
".watch-history-16"
".monitoring-beats-mb"
".monitoring-kibana-mb"
"synthetics"
".ml-state"
"ilm-history"
".monitoring-es-mb"
".monitoring-logstash-mb"
".slm-history"
".ml-anomalies-"
"metrics"
".ml-notifications-000002"
".deprecation-indexing-template"
"logs"
".ml-stats"

Most of these are data streams. I think we should discuss how best to model upgrades to these templates. I am not sure if I agree with using a version number in the name of the template especially if it also requires a new data stream name. Some access patterns may have the full data stream name hard coded and we could break those access patterns (unless we introduced aliases which isn't quite 100% for data streams). Ideally the version number would be only an internal implementation detail.

[andreidan]

I am not sure if I agree with using a version number in the name of the template especially if it also requires a new data stream name.

@jakelandis changing the data stream name wouldn't be required. When creating the newly named index template we'd need to use a priority higher than the one used by the index template we're retiring (and eventually, a bit of future thinking, deleting by the same infrastructure).

[andreidan]

@jakelandis Having done some more testing I agree we need to discuss this.

You're right that we'd generate a new data stream name, but not because we'd use the version in the index template name, but because we use the version in the index pattern of the index template already https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/ilm-history.json#L3

In other words, any update to the index template for which we'd normally bump the version will generate a new data stream name. And this is by design. And we should indeed revise this decision (I'm adding the team-discuss label for this).

[andreidan]

We discussed this today and decided to introduce an internal version for the data stream name that will mostly remain stable and use the index template version for versioning the index template. We'll also work on more guidance and governing around the IndexTemplateRegistry infrastructure in a separate effort.