Decouple phase definitions from retention in ILM policies?

[jpountz]

One challenge with ILM policies is that knowledge that is required to write a good ILM policy is scattered across different parties. On the one hand, some aspects of ILM policies are tied to the data, to the point that we'd like to bind them to our integrations. But on the other hand, end users are the only ones who know how much retention they want, and how they would like to trade performance for costs.

Here are some examples:

• For datasets that are naturally append-only and don't contain sensitive information (e.g. nginx.access), we could use the searchable_snapshots action during rollover instead of waiting for the Cold phase, resulting in better cost efficiency.
• While rollups are popular for metrics, some datasets that have high-cardinality dimensions (e.g. the process ID dimension of the system.process dataset) aren't good fits for rollups.

Could we somehow decouple phase definition from retention in ILM policies, so that integrations could own phase definitions while users would still have control on the performance/cost trade-off by owning how long data remains in each phase?

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[joegallo]

I'm removing the team-discuss label from some older Team:Data Management issues -- we've had plenty of time to discuss them, but we haven't, so the label isn't serving its purpose. Feel free to delete this comment and/or re-add the team-discuss label.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)