[Rollover] Autodelete Empty Indices

[pickypg]

It would be nice if rollover supported automatically deleting the previous index if it is empty after the alias shuffle, either via Rollover or ILM.

While checking up on one of the clusters that I maintain, I came across this subset of relevant indices:

GET /_cat/indices?s=index&v
health status index                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana-event-log-7.10.2-000002           tNQdP5bOTjyHbL9C7_YuAQ   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.10.2-000003           tSi-YQk-TaK4xVkHtkVy9A   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.10.2-000004           KSEY5pKZThanhBoqWnVNNQ   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.10.2-000005           iDLCky7RSLCnjgfPJ5GRAg   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.11.1-000001           4gou9hb5Tiu9Rq6DujOyWg   1   1          2            0       22kb           11kb
green  open   .kibana-event-log-7.11.1-000002           SQv81FipTT23j-iEAJcEbw   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.11.1-000003           q--AmEXsQWata0Z3Kc-2ag   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.11.2-000001           w0dPIouTQrWQHGy8UtM_1w   1   1          1            0     11.2kb          5.6kb
green  open   .kibana-event-log-7.11.2-000002           4oKVgQceSYOxfygc2B6TRg   1   1          1            0     11.2kb          5.6kb
green  open   .kibana-event-log-7.11.2-000003           oWnGt5tGT32utM5hPq_Ung   1   1          0            0       416b           208b
green  open   .kibana-event-log-7.12.1-000001           Y47ozbrwQ8uviGAcqeeTgA   1   1          1            0     11.2kb          5.6kb
green  open   .siem-signals-default-000001              JSTo6iseR16BuBWYzTMdLQ   1   1          0            0       416b           208b
green  open   .siem-signals-default-000002              VsA06tCaQDS1gxAqPdYqxw   1   1          0            0       416b           208b
green  open   .siem-signals-default-000003              LXO6_7dTRzGJu-Xioy-55w   1   1          0            0       416b           208b
green  open   .siem-signals-default-000004              kGmZH2x_ReSQbsF0Cd9-0g   1   1          0            0       416b           208b
green  open   .slm-history-1-000003                     HNqLm-4ySEKCQ7uinKYadQ   1   1          0            0       522b           261b
green  open   .slm-history-3-000005                     5nHdf_p-QGq-5rYaH_fvPg   1   1          0            0       416b           208b
green  open   ilm-history-1-000003                      7AUX24SxRZGCmEm9mpWEcA   1   1          0            0       522b           261b
green  open   ilm-history-3-000005                      QMbq9-EWQxKgbJUVaCTfkw   1   1          0            0       416b           208b

Note that the vast majority of the non-latest indexes are empty.

There's a clear pattern of internal indices (there's obviously nothing special about internal indices that triggers this behavior) where they leave a bunch of empty indices behind due to time. It's not necessarily a bad idea to rollover due to time, regardless of size, to take advantage of ES upgrades and template changes, but leaving around empty indices is clearly wasteful and unhelpful.

In the above example, there are more empty indices than there are documents. With an autodelete feature, there would be at-most 12 indices versus the 19 that exist. It would be even more ideal if there was a system in place to help the index version names (e.g., history-1 to history-3) that was also understood by ILM / Rollover to avoid those unneeded transitionary indices as well.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[RobDTech]

This would be such a great feature. In the meantime could it be done with a watcher? Deleting indices in warm with zero documents

[pickypg]

This would be such a great feature. In the meantime could it be done with a watcher? Deleting indices in warm with zero documents

Yes. Watches can be scripted to poll pretty much anything, then perform arbitrary actions (like a DELETE request).