[CI] NativePrivilegeStoreCacheTests testRolesCacheIsClearedWhenPrivilegesIsChanged failing

[n1v0lg]

The error message:

action [cluster:monitor/health] is unauthorized for user [test_role_cache_user] with effective roles [] (assigned roles [test_role_cache_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]

Points to test_role_cache_role role missing, which is odd because we create it as part of the test setup.

What also seems weird is that the test does not reproduce locally so I'm guessing this is a race condition around role creation.

Build scan: https://gradle-enterprise.elastic.co/s/zb7dzgsmlz6vq/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests/testRolesCacheIsClearedWhenPrivilegesIsChanged

Reproduction line:

./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests.testRolesCacheIsClearedWhenPrivilegesIsChanged" -Dtests.seed=1A83D506A6DBC37C -Dtests.locale=sr-Latn-ME -Dtests.timezone=Asia/Bahrain -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches: main

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests&tests.test=testRolesCacheIsClearedWhenPrivilegesIsChanged

Failure excerpt:

org.elasticsearch.ElasticsearchSecurityException: action [cluster:monitor/health] is unauthorized for user [test_role_cache_user] with effective roles [] (assigned roles [test_role_cache_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]

  at __randomizedtesting.SeedInfo.seed([1A83D506A6DBC37C:8EFD4ADA8802EF6D]:0)
  at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:36)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:949)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.actionDenied(AuthorizationService.java:926)
  at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:1005)
  at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:991)
  at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:952)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$7(AuthorizationService.java:447)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:187)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:437)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:413)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:314)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:149)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:201)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$buildThenMaybeCacheRole$7(CompositeRolesStore.java:369)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromDescriptors(CompositeRolesStore.java:427)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildThenMaybeCacheRole(CompositeRolesStore.java:350)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$buildRoleFromRoleReference$4(CompositeRolesStore.java:288)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.RoleDescriptorStore.lambda$resolveRoleNames$3(RoleDescriptorStore.java:171)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.RoleDescriptorStore.lambda$loadRoleDescriptorsAsync$8(RoleDescriptorStore.java:233)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:132)
  at org.elasticsearch.xpack.security.authz.store.RoleDescriptorStore.lambda$loadRoleDescriptorsAsync$12(RoleDescriptorStore.java:260)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.NativeRolesStore$4.onResponse(NativeRolesStore.java:377)
  at org.elasticsearch.xpack.security.authz.store.NativeRolesStore$4.onResponse(NativeRolesStore.java:373)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:160)
  at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:211)
  at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165)
  at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250)
  at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)
  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1367)
  at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1466)
  at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1437)
  at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41)
  at org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:38)
  at org.elasticsearch.action.ActionListener.run(ActionListener.java:567)
  at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:38)
  at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:20)
  at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
  at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
  at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.lang.Thread.run(Thread.java:833)

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[n1v0lg]

Oddly enough it looks like it used to fail a year or so ago in a similar way on 8.0:

https://github.com/elastic/elasticsearch/issues/83140

That issue was closed since the failure only occurred on 8.0

[davidkyle]

Failures on main an 8.7, mute incoming

https://gradle-enterprise.elastic.co/s/4d3s6o27awszi/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests/testRolesCacheIsClearedWhenPrivilegesIsChanged?top-execution=1

[elasticsearchmachine]

This issue has been closed because it has been open for too long with no activity.

Any muted tests that were associated with this issue have been unmuted.

If the tests begin failing again, a new issue will be opened, and they may be muted again.

[slobodanadamovic]

Any muted tests that were associated with this issue have been unmuted.

Reopening because the test is still muted.