search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.56k stars 24.35k forks source link

action [indices:admin/settings/update] is unauthorized for user [remote_monitoring_user] #102620

Open abitmore opened 7 months ago

abitmore commented 7 months ago

Elasticsearch Version

8.10.1 / 8.11.1

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu 20.04 LTS

Problem Description

On 2023-10-04 I installed MetricBeat 8.10.2 for my ElasticSearch 8.10.1 instance. All were running fine.

Starting from 2023-11-03 errors appear in elasticsearch.log (see logs below).

action [indices:admin/settings/update] is unauthorized for user [remote_monitoring_user] with effective roles [remote_monitoring_agent,remote_monitoring_collector] on indices [.ds-metricbeat-8.10.2-2023.10.04-000001], this action is granted by the index privileges [manage,all]

On 2023-11-14 I upgraded ElasticSearch, Kibana and MetricBeat to version 8.11.1, and the same errors still appear every 10 minutes.

Kibana screenshots related to the error:

image

image

Steps to Reproduce

N/A

Logs (if relevant)

[2023-11-03T13:31:38,141][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.10.04-000001] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [metricbeat]
[2023-11-03T13:31:38,294][INFO ][o.e.c.m.MetadataCreateIndexService] [bts7d-172-31-64-183] [.ds-metricbeat-8.10.2-2023.11.03-000002] creating index, cause [rollover_data_stream], templates [metricbeat-8.10.2], shards [1]/[1]
[2023-11-03T13:31:38,411][INFO ][o.e.i.m.MapperService    ] [bts7d-172-31-64-183] [.ds-metricbeat-8.10.2-2023.11.03-000002] reloading search analyzers
[2023-11-03T13:31:38,416][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.11.03-000002] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [metricbeat]
[2023-11-03T13:31:38,416][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.10.04-000001] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [metricbeat]
[2023-11-03T13:31:38,437][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.11.03-000002] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [metricbeat]
[2023-11-03T13:31:38,478][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.11.03-000002] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [metricbeat]
[2023-11-03T13:31:38,478][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.10.04-000001] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [metricbeat]
[2023-11-03T13:31:38,479][INFO ][o.e.x.i.IndexLifecycleTransition] [bts7d-172-31-64-183] moving index [.ds-metricbeat-8.10.2-2023.10.04-000001] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [metricbeat]
[2023-11-03T13:31:38,502][ERROR][o.e.x.i.IndexLifecycleRunner] [bts7d-172-31-64-183] policy [metricbeat] for index [.ds-metricbeat-8.10.2-2023.10.04-000001] failed on step [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}]. Moving to ERROR step
org.elasticsearch.ElasticsearchSecurityException: action [indices:admin/settings/update] is unauthorized for user [remote_monitoring_user] with effective roles [remote_monitoring_agent,remote_monitoring_collector] on indices [.ds-metricbeat-8.10.2-2023.10.04-000001], this action is granted by the index privileges [manage,all]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:36) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:978) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.actionDenied(AuthorizationService.java:955) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:1034) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1020) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:981) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:397) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.1
0.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:273) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:193) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:96) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:71) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1061) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:377) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.1
0.1.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.1
0.1.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.10.1.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146) ~[?:?]

...

[2023-11-26T00:08:09,795][INFO ][o.e.x.i.IndexLifecycleRunner] [bts7d-172-31-64-183] policy [metricbeat] for index [.ds-metricbeat-8.10.2-2023.10.04-000001] on an error step due to a transient error, moving back to the failed step [set-indexing-complete] for execution. retry attempt [3231]
[2023-11-26T00:08:09,828][ERROR][o.e.x.i.IndexLifecycleRunner] [bts7d-172-31-64-183] policy [metricbeat] for index [.ds-metricbeat-8.10.2-2023.10.04-000001] failed on step [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}]. Moving to ERROR step
org.elasticsearch.ElasticsearchSecurityException: action [indices:admin/settings/update] is unauthorized for user [remote_monitoring_user] with effective roles [remote_monitoring_agent,remote_monitoring_collector] on indices [.ds-metricbeat-8.10.2-2023.10.04-000001], this action is granted by the index privileges [manage,all]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:36) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:991) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.actionDenied(AuthorizationService.java:968) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:1047) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1033) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:399) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:133) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:108) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1074) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:379) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]

...
elasticsearchmachine commented 7 months ago

Pinging @elastic/es-security (Team:Security)