[ResponseOps][event log] unable to set "partial-*" event log indices to system index

[pmuellr]

Seen in an 8.6.0 deployment:

error setting existing ".kibana-event-log-8.2.3-000002" index aliases - error setting existing index aliases for alias .kibana-event-log-8.2.3-000002 to is_hidden: security_exception: [security_exception] Reason: action [indices:admin/aliases] is unauthorized for user [redacted] with effective roles [redacted] on indices [partial-.kibana-event-log-8.2.3-000002], this action is granted by the index privileges [manage,all]

The code this is generated from is below. It's purpose is to mark the old event log indices as "hidden" indices, which we didn't use to do, but do now. We're "fixing" old indices. But the fix is just to mark them hidden. The only downside of these failures is that the indices might be visible to users when they are selecting indices from index pickers and such.

https://github.com/elastic/kibana/blob/4c9a76cabdb6774e8f721b744cc264148c67aa40/x-pack/plugins/event_log/server/es/cluster_client_adapter.ts#L310-L342

I think the "partial-" prefix on the index is an indication the ILM policy has moved the indices to a non-hot tier. Rule registry has also run into this, not sure I'd seen it in the event log yet though.

So ... we should avoid doing this. Hopefully we can programmatically determine what to do with these indices - avoid them, or branch and do something else with them. The messages are not great though, as customers will think something bad has happened, when actually the problem is fairly minor.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)