ILM policies dictate when data rolls over from one index to another inside of the hot phase, and also when data transitions from one phase to another. Users will benefit from seeing this transition timing information in the ILM UI so that they can quickly form an expectation of how long their data will remain in any given phase.
If the rollover timing is strictly based on time, transition timing is relatively predictable. For example, if rollover occurs after 5 days and data transitions from the hot phase to the cold phase 10 days after rollover, then one can add the rollover timing to the transition timing and therefore predict that data will be in the hot phase for a total of 15 days.
However, rollover can be determined by sizing criteria in addition to timing criteria. For example, rollover could be configured to occur after 5 days or after 50gb, whichever occurs first. If the transition timing is still 10 days, then data will be in the hot phase for between 10 and 15 days, but the precise timing is impossible to predict.
The problem
If we surface timing in the ILM UI, it's critical that we communicate the margin of error clearly and provide users with access to the deeper implementation-level information that explains the reason for this margin of error, whether in the UI or in the docs.
We originally explored surfacing timing in the ILM UI and then decided to remove it (https://github.com/elastic/kibana/pull/90004) because of concerns about how well we were addressing these concerns. In order to reintroduce this timing information in the UI, we need a high level of confidence that we're not making things confusing or deceptively over-simplified for users.
Here's how the summary currently looks:
Rough ideas and next steps
Let's explore different options for surfacing timing in the policy summary. For example, one suggestion from https://github.com/elastic/kibana/issues/89825 was to include rollover timing in the transition timing. Here are some examples of how policies configured in different ways could be represented.
Policy with only time-based rollover
For a policy with rollover after 5 days and the hot->warm phase transition after 10 days, the summary would look like this:
Hot phase
Warm phase
15 days
Forever
Policy with both time-based and size-based rollover
For a policy with rollover after 5 days or 50gb and the hot->warm phase transition after 10 days, the summary would look like this:
Hot phase
Warm phase
10-15 days (i)
Forever
The (i) represents a tooltip that explains: "Hot phase rollover is configured with variable timing. See the hot phase for more info." Here's how the summary would look:
In the hot phase, the rollover section could offer either a link to docs about how rollover works or offer a button for opening up in-app docs. In either case, the user will have access to content that explains how size-based rollover could cause rollover to occur at any time, independent of the rollover's timing configuration. The UI already offers the link to docs:
Background
ILM policies dictate when data rolls over from one index to another inside of the hot phase, and also when data transitions from one phase to another. Users will benefit from seeing this transition timing information in the ILM UI so that they can quickly form an expectation of how long their data will remain in any given phase.
If the rollover timing is strictly based on time, transition timing is relatively predictable. For example, if rollover occurs after 5 days and data transitions from the hot phase to the cold phase 10 days after rollover, then one can add the rollover timing to the transition timing and therefore predict that data will be in the hot phase for a total of 15 days.
However, rollover can be determined by sizing criteria in addition to timing criteria. For example, rollover could be configured to occur after 5 days or after 50gb, whichever occurs first. If the transition timing is still 10 days, then data will be in the hot phase for between 10 and 15 days, but the precise timing is impossible to predict.
The problem
If we surface timing in the ILM UI, it's critical that we communicate the margin of error clearly and provide users with access to the deeper implementation-level information that explains the reason for this margin of error, whether in the UI or in the docs.
We originally explored surfacing timing in the ILM UI and then decided to remove it (https://github.com/elastic/kibana/pull/90004) because of concerns about how well we were addressing these concerns. In order to reintroduce this timing information in the UI, we need a high level of confidence that we're not making things confusing or deceptively over-simplified for users.
Here's how the summary currently looks:
Rough ideas and next steps
Let's explore different options for surfacing timing in the policy summary. For example, one suggestion from https://github.com/elastic/kibana/issues/89825 was to include rollover timing in the transition timing. Here are some examples of how policies configured in different ways could be represented.
Policy with only time-based rollover
For a policy with rollover after 5 days and the hot->warm phase transition after 10 days, the summary would look like this:
Policy with both time-based and size-based rollover
For a policy with rollover after 5 days or 50gb and the hot->warm phase transition after 10 days, the summary would look like this:
The (i) represents a tooltip that explains: "Hot phase rollover is configured with variable timing. See the hot phase for more info." Here's how the summary would look:
In the hot phase, the rollover section could offer either a link to docs about how rollover works or offer a button for opening up in-app docs. In either case, the user will have access to content that explains how size-based rollover could cause rollover to occur at any time, independent of the rollover's timing configuration. The UI already offers the link to docs: