search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.63k stars 24.64k forks source link

Introduce the notion of system composable index templates #65957

Open martijnvg opened 3 years ago

martijnvg commented 3 years ago

Currently many index template and composable index template exist for Elastic (internal) features, these templates are installed by the elected master node when the cluster has booted. If these templates are removed then these templates are added back and there is logic that optionally updates these templates if after an Elasticsearch upgrade.

Besides what is described above, there isn't anything special about these template. While these templates should be treated in a special way. For example these templates shouldn't be returned by default in the get template api. Nor should it be possible to even attempt to delete or modify those templates.

I think it would be great if system composable index templates and system component templates could be defined in the same way as system indices (Either via SystemIndexPlugin interface or another plugin interface). This should replace IndexTemplateRegistry and subclasses.

elasticmachine commented 3 years ago

Pinging @elastic/es-core-features (Team:Core/Features)

gwbrown commented 3 years ago

It sounds like this overlaps substantially with https://github.com/elastic/elasticsearch/pull/65604, we should make sure we're coordinating efforts.

/cc @pugnascotia, @jaymode

martijnvg commented 3 years ago

Thanks @gwbrown for referencing that pr. I wasn't sure whether this falls under system indices, but the goal for opening this issue was for improving how we today manage templates of internal data streams / time based index patterns, which currently aren't a system indices, but are just hidden, which seems to fall outside the scope of the current system index development effort.

pugnascotia commented 3 years ago

There are obvious parallels between what I'm doing for system indices and the index template registry. However as the moment, I don't believe any system indices strictly require a template, because we can embed the settings and mappings in the system index descriptor. There's no support right now if a system index definitely requires a template - for example, if we ever wanted a system data stream. I had thought that a system data stream was unlikely.

At the moment, I think any implementation for this issue would be similar to #65604, but wouldn't actually overlap per se.

martijnvg commented 3 years ago

@pugnascotia I agree with your assessment that system indices have no dependency on this issue which purpose is to replace template registry infrastructure with system index templates.