"cluster:admin/snapshot/delete" not allowed for kibana user for apm blocks deletes of ILM searchable snapshot

[stefnestor]

👋🏽 howdy, team! (Marking "Security" as the team majorly editing the file but relates to security during ILM actions.)

It appears ESS file-based user: found-internal-kibana4-server covers ES's kibana service account with role: kibana_system. AFACT part of this role is defined here where the Kibana user is granted permissions against various APM/Synthetics indices in order to progress them through ILM deletion. However, where ILM policies enable Searchable Snapshots, this role is unable to run cluster:admin/snapshot/delete so data never successfully deletes.

This induces a permanent-until-intervention ILM deletion error

Oct 10, 2024 @ 20:34:48.247 [instance-0000000065] policy [metrics-apm.app_metrics-default_policy] for index [restored-.ds-metrics-apm.app.asdf-default-2024.04.16-000015] failed on step [{""phase"":""delete"",""action"":""delete"",""name"":""cleanup-snapshot""}]. Moving to ERROR step
org.elasticsearch.ElasticsearchSecurityException: 
    action [cluster:admin/snapshot/delete] is unauthorized for user [found-internal-kibana4-server] with effective roles [found-internal-kibana4-server,kibana_system], this action is granted by the cluster privileges [manage,all]

Due to role permissions

$ cat kibana_system.json | jq -rc '.kibana_system.indices[]|select(.names[]|contains("metrics-apm"))|{names:.names, privileges:.privileges}' | uniq
{"names":["metrics-apm.*"],"privileges":["read","read_cross_cluster"]}
{"names":[".logs-endpoint.diagnostic.collection-*","logs-apm-*","logs-apm.*-*","metrics-apm-*","metrics-apm.*-*","traces-apm-*","traces-apm.*-*","synthetics-http-*","synthetics-icmp-*","synthetics-tcp-*","synthetics-browser-*","synthetics-browser.network-*","synthetics-browser.screenshot-*"],"privileges":["indices:admin/delete"]}

AFAICT this should affect APM-or-Synthetics any time they encounter Searchable Snapshots where ILM Delete has delete_searchable_snapshot: true.

For more verbose example, kindly see internal details for v(8.13.4, 8.14.2) but it doesn't appear code has changed since.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

Relates to #72856