Better rollover handling of small and empty indices

[joegallo]

This meta issue tracks some enhancements and other changes around handling rollovers (especially ILM rollovers) better in the cases of small or empty indices.

• Enhancements

  • [x] https://github.com/elastic/elasticsearch/pull/83345
  • [x] https://github.com/elastic/elasticsearch/issues/86203

• Related changes

  • [ ] https://github.com/elastic/kibana/issues/137464
  • [ ] https://github.com/elastic/elasticsearch/issues/89284
  • [ ] Evaluate adding a min_* condition by default to stack-included ILM policies (e.g. ilm-history-ilm-policy, logs, as well as the policies installed by APM or Kibana, etc.)
  • [ ] https://github.com/elastic/elasticsearch/issues/83039 and just a little more broadly, documenting the idea of how to retire an index alias or datastream around an ILM policy

• Tech Debt

  • [ ] https://github.com/elastic/elasticsearch/issues/88018
  • [ ] https://github.com/elastic/elasticsearch/issues/89282

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[joegallo]

re: https://github.com/elastic/elasticsearch/issues/83039 (that we don't have an especially great way to 'retire' an alias or datastream that's not going to be written to anymore):

At the moment we have some error conditional handling around situations where the index in question is unexpectedly marked with index.lifecycle.indexing_complete set to true:

https://github.com/elastic/elasticsearch/blob/1a3032beb6790805b2bba01180f9b875ac2fa971/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/WaitForRolloverReadyStep.java#L130-L168

A potentially interesting implementation of alias/datastream 'retirement' would be to revise that logic a bit so that we could merely mark an index as finished by setting index.lifecycle.indexing_complete to true and then having ILM continue past check-rollover-ready and continue on through the rest of the lifecycle. In order to make this work, though, we'd need to juggle is_write_index when we see this (since the rollover itself wouldn't handle that anymore). Taking that a step further, for datastreams that would open up the possibility that a datastream would have no current write index, so a nearby or parallel effort to this would be having the ability to make a datastream read only.