search

elastic / apm-server

https://www.elastic.co/guide/en/apm/guide/current/index.html
Other
1.22k stars 525 forks source link

Automatically migrate default ILM policy to Data Stream Lifecycle Management #14128

Open axw opened 2 months ago

axw commented 2 months ago

In 8.15.0 we enable the apm-data plugin by default in Elasticsearch, which ships index templates that make use of data stream lifecycle by default. It is still possible to opt into ILM (except in Serverless projects, which do not support ILM).

There was a bug in 8.15.0 that we fixed in 8.15.1 (https://github.com/elastic/apm-server/issues/13898): data streams upgraded from older versions would be "unmanaged" on upgrade. Data stream lifecycle is only automatically configured on newly data streams, so we need to keep some ILM configuration for existing data streams.

This leaves us in an indefinite undesirable state:

We should automatically migrate existing data streams from ILM to data stream lifecycle management when the existing data streams use the default ILM policy. The retention durations are aligned, and the default ILM policy does not use any advanced functionality (e.g. hot/warm/cold tiering), so this should be considered a non-breaking change.

pilsy commented 2 months ago

It's not really an enhancement, it's more of a bug -- how do i migrate this myself? none of the tracing stuff works anymore in our instances... (we are spending 25k a year on them, and they no longer work)

axw commented 2 months ago

@pilsy which part's a bug? There's definitely a bug in 8.15.0, for which there's a fix in 8.15.1: https://github.com/elastic/apm-server/issues/13898

Is there something else affecting you too?

pilsy commented 1 month ago

@axw support eventually reached out and helped me fix the problem, basically when the cluster was upgraded to 8.15.0 all tracing, logging and metrics stopped displaying in the interface, even tho the data was still being ingested... and a simple upgrade to 8.15.1 did not automagically fix it.

axw commented 1 month ago

@pilsy thanks, I will follow up on the support case and see what else we can/need to do. Sorry about the trouble this caused you.

EDIT: I see the resolution was to force a rollover to the data streams. This is related to https://github.com/elastic/elasticsearch/issues/112781