[REQUEST]: Discourage cloning index templates - offer easier option to customize settings if not using namespaces + Warning/Disclaimer page

[lucabelluccini]

Description

We're aware of the issues cloning index templates of Fleet integration leads to.

We need to attempt to warn users / push users to not use index template cloning or edit the index & component templates.

The page https://www.elastic.co/guide/en/fleet/current/data-streams-ilm-tutorial.html should be reviewed to display 2 options:

• one in case the user wants to edit the ILM policy (this would involve creating a @custom component template to override the ILM policy or edit the default ILM policy coming with the integration) - point to https://www.elastic.co/guide/en/fleet/current/data-streams.html#data-streams-ilm
• one in case the user wants to have a namespace specific ILM policy (this will require index template cloning)
  • for the namespace specific customizations, we need to point to a new "warnings/disclaimer" page

Further enhance https://www.elastic.co/guide/en/fleet/current/integrations-assets-best-practices.html

• Never edit managed assets of Fleet (e.g. the Index template, the @package component templates and/or ingest pipelines bundled with integrations). Any change will be overwritten upon upgrade/install.

• When the user creates a @custom component template for a package integration, while allowed and legit, the user can incur in some risks which might prevent the correct ingestion of data.

  • It can lead to broken indexing, data loss, broken integration packages upgrades.
    • Example: the @package component template of the integration switches from a "normal" data stream to TSDB or LogsDB and some settings/mappings introduced by the user are not compatible with such indexing modes.
    • Example: the user overrides the type of an ECS field from keyword to text - this might prevent the aggregation on such field on built-in dashboards.
    • We warn about this on this page

• When users create a @custom ingest pipeline as we document here, Elastic is not responsible of the correct indexing/behavior of the ingest pipeline. It becomes a custom processing of the data and it's user responsibility to test it.

  • We might add the warning here

• When a user clones the index template of an integration package, the user can incur in the risk of not picking up index template major changes upon upgrades. The "structure" of the index template will be "frozen" at the moment it was cloned.

  • It can lead to broken indexing, data loss, broken integration packages upgrades - because the structure of the Index templates which ship with Fleet can change over minor version and the user might lag behind those changes or pick up only a part of them.
  • Covered in https://github.com/elastic/elasticsearch/issues/97664#issuecomment-2269220278 and ingest-dev/3617
  • This is "problematic" as we document to clone the index template at this page...

• Cloning index templates to add/inject additional component templates cannot be tested by Elastic, so we cannot guarantee it can work in future release and it can break at any minor release.

Resources

Referenced above

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @lucabelluccini

I can help with a draft.

[lucabelluccini]

FYI @jsoriano / @kilfoyle

Some of those warnings are already in the docs - but we need to start to better frame the message and raise more warnings.

[kilfoyle]

@lucabelluccini to get us started, I've opened this PR to add the new warnings you suggest for the Best practices for integration assets page. Please let me know whatever you think we should change.

For the first part, about updating the Customize data retention policies tutorial with two different options, I haven't worked much with component templates, etc., and would really appreciate if you or anyone can suggest exactly what steps should be added/changed.