search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.49k stars 24.6k forks source link

Ensure the templates bundled with elasticsearch are present before indexing #101768

Open gmarouli opened 10 months ago

gmarouli commented 10 months ago

Elasticsearch Version

8.11

Installed Plugins

No response

Java Version

bundled

OS Version

not-relevant

Problem Description

When elasticsearch boots up for the first time it is installing the logs template. Currently, this is an async action and it is possible for the user to start indexing before the logs composable index template is in place.

The first document to be indexed will create the index/data stream that template is configuring, but if the template is not there, the request will create an index with the default setting and mappings which is not what our users expected.

This does not happen very often in the "wild" but it is possible and it has been observed in our tests, we had to add a "wait for logs" method (https://github.com/elastic/elasticsearch/issues/99764).

Proposal: We would like to ensure that at least a version of the templates bundled with elasticsearch are present before a user can start indexing.

Steps to Reproduce

Run the original yaml tests, such as https://github.com/elastic/elasticsearch/issues/99764

Logs (if relevant)

No response

elasticsearchmachine commented 10 months ago

Pinging @elastic/es-data-management (Team:Data Management)

felixbarny commented 10 months ago

Related: https://github.com/elastic/elasticsearch/issues/97032

eyalkoren commented 9 months ago

101872 proposes one way to deal with this problem through indexing APIs.

An already existing option to work around this issue from the client side is to start with an explicit create data stream request, which will fail if there is no matching index template ready.

bhperry commented 1 month ago

It would be really great if this could be handled in a general way for any index template, not just bundled templates. I have a custom template that is applied when elasticsearch is installed, but fluent-bit starts up at the same time and will generally create the first index before the template exists. Seems like a huge gap for ES to not have a ready-made solution for this instead of requiring clients to handle it.

I get that ES cluster setup usually only happens once so it's not a heavily prioritized thing, but my company stands up new clusters all the time for customers. So a pain point that happens once every time a new cluster is created is not insignificant.