search

elastic / apm-server

https://www.elastic.co/guide/en/apm/guide/current/index.html
Other
1.21k stars 515 forks source link

Kibana does not remove index template for removed data streams on integrations upgrade #13181

Open endorama opened 3 months ago

endorama commented 3 months ago

When we will migrate from apm integration package to apm-data ES plugin we will need to trigger a removal of index templates created by the integration, which by default are created with a higher priority than the one created by the ES plugin.

To ensure this steps works I run a test to verify if Kibana removes index template when upgrading an integration where installed data streams have been removed. I tested these steps:

  1. setup a Es+Kibana stack and install apm integration package (tested with 8.15.0-SNAPSHOT but any current version should lead to same results)
  2. check installed Index Templates
  3. install a custom version of apm integration package where all data streams except one (traces - due to how elastic-package works is not possible to package an integration without at least 1 data stream; this is not relevant for this test but should be investigated to proceed further with the migration) have been removed
  4. Verify if Index Templates for removed data streams have been removed.

The test showed that Index Templates are not removed. This is a blocker for https://github.com/elastic/apm-server/issues/11529 as templates from the plugin would be overridden by the integration package templates.

Assets difference between packages Assets with normal integration package: ![2024-05-16_16-06](https://github.com/elastic/apm-server/assets/526307/1c366629-20be-4577-a4a3-a3311fa387d1) Assets with custom integration package: ![2024-05-16_16-08](https://github.com/elastic/apm-server/assets/526307/63fcaafd-7a17-4ede-b7f1-9ac19c9dfb3a)
Index templates before installing custom version ![image](https://github.com/elastic/apm-server/assets/526307/e750453b-6490-4835-a1b5-2abe070d3932) ![image](https://github.com/elastic/apm-server/assets/526307/a6c92c14-5d8e-46d1-ba28-fb618e00eee2)
Index templates after installing custom version ![Screenshot 2024-05-16 at 16-09-02 Index Management - Elastic](https://github.com/elastic/apm-server/assets/526307/ad77b604-cfec-4119-a51d-0f26f30f3ef0) ![Screenshot 2024-05-16 at 16-09-10 Index Management - Elastic](https://github.com/elastic/apm-server/assets/526307/49f9c80c-15be-42c8-a777-bcc771409890)
axw commented 3 months ago

The test showed that Index Templates are not removed. This is a blocker for https://github.com/elastic/apm-server/issues/11529 as templates from the plugin would be overridden by the integration package templates.

We can unblock ourselves by increasing the priority of the templates in the apm-data plugin: https://github.com/elastic/apm-server/issues/11529#issuecomment-2119519247

Not ideal, but I think that's likely the most expedient approach. We should still go ahead with removing the data streams from the integration package, the existing templates just won't be removed. If need be, we could add some code to Kibana to remove them for APM specifically.

endorama commented 3 months ago

We moved forward with increasing the priorities in https://github.com/elastic/elasticsearch/pull/108885

I'd keep this issue open to track this Kibana behavior which may still affect the migration (es removing old templates is a manual step).