How are daily indices supposed to work with this lib?

[mgrundkoetter]

I have a logging usecase in a Symfony environment. When I call $indexBuilder->createIndex('my-index') it creates an index with a suffix like this my-index_2023-09-18-105330. When I now call $indexer->scheduleIndex('my-index', $doc) it creates another index my-index and puts the content there, not into "my-index_2023-09-18-105330" as expected. On the other hand when I call $client->getIndex('my-index') to provide this to the scheduleIndex method as first param, I also get the 'my-index' directly, not the my-index_2023-09-18-105330 one.

What I actually wanted is an index like my-index-YYYY-mm-dd (without time), but I can not configure it this way :-( And I also want to index new documents into the index with the current date and have an alias with the "clean" name without the suffix. I can't find anything in the docs, so I will try it here: How can this be done with this library?

[damienalexandre]

When I now call $indexer->scheduleIndex('my-index', $doc) it creates another index my-index and puts the content there

This is not supposed to happen, as my-index should be an alias, not an index.

My supposition is that you are not calling this code after creating your indexes:

// Set the proper aliases
$indexBuilder->markAsLive($index, 'my-index');

Would you mind sharing your implementation?

What I actually wanted is an index like my-index-YYYY-mm-dd (without time), but I can not configure it this way :-(

That's right, the index name format is deeply hard-coded because it affect a lot of logic (index retention, alias name retrieval...).

[mgrundkoetter]

I create the index but when I do $indexBuilder->markAsLive($this->client->getIndex('my-index'), 'my-alias'); afterwards, I get the error

[ERROR] no such index [my-alias] [index: my-alias]

If the format is hardcoded, how is it supposed to work to store multiple events in the same index if every store action would trigger a new index in the predefined date format? When I do $client->getIndex('my-index') it will not return my Index as the name is actually suffixed.

I get several hundred thousands log events a day and I want one index per day (For ILM with Kibana). How can I achieve this when I have no control over the actual index name in ES? I Dont want multiple days in one index nor do I want several indices per day.

[damienalexandre]

I create the index but when I do $indexBuilder->markAsLive($this->client->getIndex('my-index'), 'my-alias'); afterwards, I get the error

Yeah, you are supposed to give the brand new index created by createIndex here, as shown in the documentation.

// Create the Index in Elasticsearch
$index = $indexBuilder->createIndex('beers');

// Set the proper aliases
$indexBuilder->markAsLive($index, 'beers');

If the format is hardcoded, how is it supposed to work to store multiple events in the same index if every store action would trigger a new index in the predefined date format?

Elastically never triggers a new index, you do that on your own with createIndex.

If you have trying to store logs in daily indices, where every event is stored in the proper index where the date of the event is also the date of the index, Elastically may not be the right tool.

[mgrundkoetter]

Elastically never triggers a new index, you do that on your own with createIndex.

Yes, sure. But imagine I have already created an index and stored my first event of the day. Now the second event comes in. I could either create another index for this or try to use the one I created before. But because of the fixed naming, I have no chance to get it back out because it simply has another name than the one I provided. That's what I meant with "every store action will trigger a new index".

If you have trying to store logs in daily indices, where every event is stored in the proper index where the date of the event is also the date of the index, Elastically may not be the right tool.

Yeah, that's probably true and also kind of unexpected. In the ELK universe, that's the default logic for events. Like in logstash where you normally use a template format for indices like %{+YYYY.MM.dd}. It also seems that rollover and datastreams are not supported with elastically as well.

[damienalexandre]

Yes Elastically is more appropriate for application indexes.

You may use https://github.com/ruflin/Elastica directly without the "application framework paint" I suppose.

I'm closing this issue, don't hesitate if you have any other question :pray: