[Integrations] Add a link to ILM policies in the integration policy editor

[mostlyjason]

A problem with our current UX for integrations is that users don't have an easy way to discover or change their data retention settings when they are adding data to Elastic. The current experience is that the user adds an integration. They then have to understand that this integration adds data streams, which have index templates and ILM policies. They also have to understand that they can change their data retention using the ILM policy in the stack management app. There is nothing in the UI helping users understand these concepts or connections.

As a near term solution, we should provide a link to documentation users them more information about how these concepts connect and how to change the settings. It should come after the step where the user selects the data inputs they want to enable, since this will determine what policies they need to set (logs, metrics, traces, etc). It should be hidden under the advanced settings box since the steps are arduous. This will provide a workaround until we build an easy UI feature.

User stories:

• As a service owner, I want to discover how to set up ILM policies when I am adding an integration for my service so that I can set my retention and data tiers
• As a service owner, I want to set ILM policies per namespace, so that I can have different polices for test and production environments, or for different teams.
• As a service owner, I want documentation on how to customize my ILM policies in a way that is forward-compatible with future package upgrades and UI features

Related:

• https://github.com/elastic/observability-docs/issues/986

Blocked by:

• [x] https://github.com/elastic/kibana/issues/121184
• [x] https://github.com/elastic/kibana/issues/121099
• [ ] https://github.com/elastic/kibana/issues/121118 - up for debate, but would simplify the documentation considerably

CC @cjcenizal @dborodyansky @jen-huang

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mostlyjason]

Mockup to help illustrate the concept (not final)

[jen-huang]

Instead of showing these links in the policy editor, have we considered showing them as part of the integration details page? For example maybe a Data retention section under the Settings tab (screenshot). I have a few reasons for this proposal:

1. If the user comes to the Fleet policy editor to configure an integration they are using for the first time, Fleet has not installed that integration yet, therefore the ILM policies that integration ships is not installed yet either so there is nowhere to link to (apart from the Stack-installed logs, metrics, synthetics ILM policies).
2. The ILM policies shipped by an integration apply to all incoming data streams for that integration, regardless of the namespace, which is part of what the user configures in the policy editor. In other words, the policy editor allows the user to configure a particular instance of an integration, but the ILM policies apply to all instances of an integration.

There are also one more major caveat: similar to other assets that Fleet installs (dashboards, index templates, etc), when an integration is upgraded, any user changes are overridden as we remove the previous assets and reinstall them (LOC for data stream ILM policies). This directly conflicts with the goal of adding links to allow the user to edit the ILM policies since, well, the user changes will be gone on next upgrade!

We knew of this limitation so in 7.14 we added installation of a custom component template for every index template Fleet installs. The contents of these custom component templates are not overridden at any point, so this enables users to set and persist custom settings for their data streams, including attaching a custom ILM policy. The custom component templates are also namespace-agnostic, so all the more reason to have any Data retention UI live at the integration details level, rather than the policy editor.

I will try to find an integration that ships ILM policies so that we have a concrete case to work from for further technical definition.

[joshdover]

From our discussion this week, it appears the main use case that we're trying to solve is allow users to view (and possibly configure) the ILM policy that applies to an integration for the configured namespace. In other words, the level of granularity we're trying to achieve is integration-specific and namespace-specific. The rest of this comment assumes this is still the use case that we want to solve. If there are other levels of granularity we want to explore, other options are available. However, I don't believe there is a path forward with the current architecture for having the ability to set integration policy-specific ILM policies, because we would need to introduce integration policy-specific data streams which was not considered tenable due to exploding the number of indices in the cluster.

As Jen mentioned above, the integration policy editor is not necessarily namespace-specific. You can have multiple policies for the same integration/package that are outputting to the same namespace. We need to make sure that this is accounted for in the UI or else we risk confusing the user into thinking they're only setting the retention policy for this specific integration, rather than all integrations of this package that are targeting this namespace.

In some of the long-term UX mocks, the namespace picker is part of this UI:

However, I think we should consider several changes to make this mechanism more clear:

• We probably should show the namespace field first, since the values of the other fields depend on the user's namespace choice.
• We should make clear which other integration policies this change is going to affect if there are other integration policies of the same package that are indexing into this namespace.

Tangential to this UX issue, I also want to make clear that we don't currently have any component templates that a customer can manually modify to set the ILM policy at the integration-specific, namespace-specific level of granularity today. The @custom component templates added in 7.14 are datastream-specific (there may be multiple streams in a single integration) and apply to all namespaces. I believe this throws a wrench in being able to provide simple documentation or in-app instructions to a user on how to set this up manually. For a package that does not ship its own ILM policy, the steps required today would be:

• Create a new component template that sets the desired ILM policy
• Duplicate the logs-<package name>.<dataset> and/or metrics-<package name>.<dataset> index templates for each dataset in the package and:
  • Update the index pattern to only match the desired namespace: logs-<package name>.<dataset>-<namespace>*
  • Increase the priority to 250
  • Add the custom component template

I'd need to test further, but I believe this would be preserved across package upgrades. That is, until we provide OOTB functionality to support this. In that case we're going to need to either fail to upgrade packages that have something custom set OR we should document a well-known naming convention for any user overrides so we can preserve them when we do add this functionality.

We also still need to take into consideration the case where the package does ship its own ILM policy in the documentation since the instructions there would be a bit different (most likely just ensuring the right priority level is set on the index template).

We could simplify this quite a bit if we included as a requirement for this UX change to start adding index templates and component templates at the namespace level, rather than only at the dataset level. For example, a integration-scoped namespace-specific index template would be logs-<package name>.*-<namespace> and a component template would be logs-<package name>.*-<namespace>@custom. However, I do worry about creating these by default and exploding the number of templates a user needs to think about. I'd prefer that we create these on-demand as needed through managed service / feature in the Kibana UI.

[jen-huang]

I created https://github.com/elastic/kibana/issues/110342 for implementation of what we discussed earlier this week for 7.16 - just a link to docs from policy editor - until we have the product and technical design fully figured out for the scope in this issue.