search

opensearch-project / index-management

🗃 Automate periodic data operations, such as deleting indices at a certain age or performing a rollover at a certain size
https://opensearch.org/docs/latest/im-plugin/index/
Apache License 2.0
53 stars 112 forks source link

ISM and using timestrings for index age #312

Open adaisley opened 2 years ago

adaisley commented 2 years ago

Is your feature request related to a problem? Please describe. Currently, we use Curator to manage our Elasticsearch cluster. We're looking at migrating to Opensearch and checking through compatibility issues right now. We'd like to switch over to using ISM to manage our cluster's indexes, but we aren't quite sure it has the same functionality right now.

There are some index patterns that we use the creation time as the age of the index that we want to move to warm/delete. Most of the time though, we use the timestring on the index. This can be set using different Curator filters, such as https://www.elastic.co/guide/en/elasticsearch/client/curator/current/fe_timestring.html and https://www.elastic.co/guide/en/elasticsearch/client/curator/current/fe_use_age.html

Describe the solution you'd like When creating a Transition in ISM, under "Minimum index age" there should be an OPTIONAL option to use a timestring format which you specify, to decide on the transition trigger, instead of going off of what I believe is index creation time. image

Describe alternatives you've considered We haven't managed to get one working, but a fork of Curator that is completely compatible with Opensearch would also be nice, though perhaps that option misses the point of moving forward and using ISM.

Additional context Add any other context or screenshots about the feature request here.

dbbaughe commented 2 years ago

Do you have an example for how the index creation date vs the date string in the index name is causing a difference for you?

downsrob commented 2 years ago

This other issue https://github.com/opensearch-project/index-management/issues/65 seems to present a use case. Index creation date is reset during reindexing, but using the date string in the index name would enable transitions based on the index creation date before reindexing, as long as that original index creation date was in the index name.