x-pack/plugin/apm-data creates templates referencing ILM policies that are not created

Evesy commented 1 month ago

Elasticsearch Version

8.15.1

Installed Plugins

apm-data

Java Version

openjdk 21.0.4 2024-07-16 LTS OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)

OS Version

Rocky Linux release 9.4 (Blue Onyx)

Problem Description

When Elasticsearch is configured with xpack.apm_data.enabled to enable the creation of APM indices (to support running APM Server standalone, without needing Fleet), the index templates that are created reference ILM policies that do not exist

For example the component template traces-apm.rum-fallback@lifecycle is configured with an ILM policy of traces-apm.rum_traces-default_policy which does not exist. The setting does include _prefer_ilm: false however it seems incorrect behaviour to have references in these templates to policies that do not exist.

I've assumed this is a bug, but it might just be it's not clear the expectation of a user on how to manage ILM policies for these APM data streams

Steps to Reproduce

Spin up a barebones Elastic instance/cluster with xpack.apm_data.enabled: true in the configuration
Observe created index & component templates without matching ILM policies

Logs (if relevant)

No response

mosche commented 1 month ago

Looks like this relates to https://github.com/elastic/elasticsearch/pull/112759

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-data-management (Team:Data Management)

philipp-intelliroad commented 1 month ago

I'm not sure if this is related but the logs@settings configuration references a policy as well that does not exist:

If I look for the policy I can not find it or can only find it by clicking a few "hidden" links somewhere. When I finally reach the policy page I see a big deprecation warning.

I guess we should change the configuration here:

https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/template-resources/src/main/resources/logs%40settings.json#L6

I guess it should point to:

https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/template-resources/src/main/resources/logs%40lifecycle.json

elastic / elasticsearch