ILM: Allow delete by space

[jguay]

Describe the feature:

If you have rollover at 50Gb, you can delete based on age from rollover in 7.4.2. However the requirement for deletion may depend on disk space usage Example : need to delete oldest index when total size >= 20Tb

This would be greater value when https://github.com/elastic/elasticsearch/issues/44001 is added... The aim would be index with names "team-a" with limit of 2Tb, and index "team-b" with limit of 500Gb. When this is reached the oldest index for matching pattern gets deleted automatically

Equivalent feature in curator : https://www.elastic.co/guide/en/elasticsearch/client/curator/current/filtertype_space.html

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[fbaligand]

This feature would be great! in delete phase, be able to specify delete index action in X “rollovers”, like we can specify delete index action in X “days”. For example, if you specify to delete index “in 5 rollovers”, this will allow to remove index “index-00001” when index “index-00006” is created by rollover.

[reighnman]

This is a feature other ES backed offering have (such as Graylog) which makes it much easier to manage storage.

If I have rollover set to Xdocs/Xsize and a delete phase that only keeps the 10 latest indexes I can estimate disk consumption fairly accurately.

[fbaligand]

I agree with @reighnman: it's very useful to control disk usage and avoid full FS!

[allaboutopensource]

Describe the feature:

If you have rollover at 50Gb, you can delete based on age from rollover in 7.4.2. However the requirement for deletion may depend on disk space usage Example : need to delete oldest index when total size >= 20Tb

This would be greater value when #44001 is added... The aim would be index with names "team-a" with limit of 2Tb, and index "team-b" with limit of 500Gb. When this is reached the oldest index for matching pattern gets deleted automatically

Equivalent feature in curator : https://www.elastic.co/guide/en/elasticsearch/client/curator/current/filtertype_space.html

this feature is only to delete the indices not the actual data which is stored in ES nodes. AM i correct?

[fbaligand]

Well, when you delete an index, you delete the data inside the index.

[allaboutopensource]

so u mean to say i can delete the indices automatically using the "delete" phase of lifecycle and that will eventually delete the logs and free up the space in my ES nodes.

[fbaligand]

Yes, that is the aim of ILM delete phase.

[phlegx]

@reighnman is right. Is such a feature planned?

I'm still wondering how other people with the current ILM rollover features are able to estimate the disk space.

One quick workaround for now could be to check the disk space and if disk becomes too full to start a script that deletes all indices but last 10 or so.

[passing]

Not having this feature in ILM still forces us to use curator. Luckily curator 5.7+ is still compatible with ES 7.x, but not sure if it will still be with ES 8

[psych0d0g]

our ES cluster is pretty dynamic, there are constant changes to what creates logs, also there are index sets coming and going with diffrent settings in rollover, so a feature in ILM that deletes old indices based on cluster watermark or free disk space remeaning would be highly essential for us, is this request even considered to be implemented? asking since this issue is already pretty "old"

[marcohevi]

Echoing everyone's sentiment here. This is pretty essential to manage large clusters, specially with multiple tenants.

[tybalex]

agreed. This is something I am looking for from ILM, I am surprised this feature is still missing...

[fzyzcjy]

This is very helpful! When can it be implemented?

[ghost]

+1

Definitely need this feature added to ILM.

[noobjun]

My disk already got bombed today without this feature, please address!

[kvokka]

ES8 does not have this feature, on the other hand, it deprecated the tool which did this job.

This is a neat bonus that was discovered after the upgrade to ES 8.2

It would be great if you add this feature to ILM or at least bump the curator so it would start to work with the current ES version.

[archon810]

Another vote from me on this matter. It's puzzling to have such a basic but important feature missing, and the very tool many have been using for the job completely left behind. Please implement in ILM or update the tool.

[kvokka]

@dakrone As you rightfully noticed in the #87816 it is not appropriate to create the issue to raise the priority. But at least I know who to ask now.

In this case, may i kindly ask you to give some thoughts about this issue, or forward this issue to the right person, or share the trick which is valid for ES8 (cos curator is deprecated), or at least say that all the commenters here that we should not expect this and it should be done by 3rd party.

Thank you!

[aivinog1]

Hello! I would like to tackle this issue, but I can't give any promises 🙂 If it is okay, I will work on this one

[petiepooo]

Still a much-requested issue. See also #100929

[passing]

If you have rollover at 50Gb, you can delete based on age from rollover in 7.4.2. However the requirement for deletion may depend on disk space usage Example : need to delete oldest index when total size >= 20Tb

We have created a python script as a workaround to solve this problem. It can be configured to limit the disk usage for all indexes in an ILM phase and will simply move the oldest index to the next phase whenever the limit is exceeded. It is available in the zooplus/ilm-limiter repository.