search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.71k stars 24.67k forks source link

[CI] NativePrivilegeStoreCacheTests testRolesCacheIsClearedWhenPrivilegesIsChanged failing #93447

Open n1v0lg opened 1 year ago

n1v0lg commented 1 year ago

The error message:

action [cluster:monitor/health] is unauthorized for user [test_role_cache_user] with effective roles [] (assigned roles [test_role_cache_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]

Points to test_role_cache_role role missing, which is odd because we create it as part of the test setup.

What also seems weird is that the test does not reproduce locally so I'm guessing this is a race condition around role creation.

Build scan: https://gradle-enterprise.elastic.co/s/zb7dzgsmlz6vq/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests/testRolesCacheIsClearedWhenPrivilegesIsChanged

Reproduction line:

./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests.testRolesCacheIsClearedWhenPrivilegesIsChanged" -Dtests.seed=1A83D506A6DBC37C -Dtests.locale=sr-Latn-ME -Dtests.timezone=Asia/Bahrain -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches: main

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests&tests.test=testRolesCacheIsClearedWhenPrivilegesIsChanged

Failure excerpt:

org.elasticsearch.ElasticsearchSecurityException: action [cluster:monitor/health] is unauthorized for user [test_role_cache_user] with effective roles [] (assigned roles [test_role_cache_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]

  at __randomizedtesting.SeedInfo.seed([1A83D506A6DBC37C:8EFD4ADA8802EF6D]:0)
  at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:36)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:949)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.actionDenied(AuthorizationService.java:926)
  at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:1005)
  at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:991)
  at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:952)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$7(AuthorizationService.java:447)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:187)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:437)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:413)
  at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:314)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:149)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:201)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$buildThenMaybeCacheRole$7(CompositeRolesStore.java:369)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromDescriptors(CompositeRolesStore.java:427)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildThenMaybeCacheRole(CompositeRolesStore.java:350)
  at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$buildRoleFromRoleReference$4(CompositeRolesStore.java:288)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.RoleDescriptorStore.lambda$resolveRoleNames$3(RoleDescriptorStore.java:171)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.RoleDescriptorStore.lambda$loadRoleDescriptorsAsync$8(RoleDescriptorStore.java:233)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:132)
  at org.elasticsearch.xpack.security.authz.store.RoleDescriptorStore.lambda$loadRoleDescriptorsAsync$12(RoleDescriptorStore.java:260)
  at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:167)
  at org.elasticsearch.xpack.security.authz.store.NativeRolesStore$4.onResponse(NativeRolesStore.java:377)
  at org.elasticsearch.xpack.security.authz.store.NativeRolesStore$4.onResponse(NativeRolesStore.java:373)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:160)
  at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:211)
  at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)
  at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165)
  at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:250)
  at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:43)
  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1367)
  at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1466)
  at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1437)
  at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41)
  at org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:38)
  at org.elasticsearch.action.ActionListener.run(ActionListener.java:567)
  at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:38)
  at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:20)
  at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
  at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
  at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.lang.Thread.run(Thread.java:833)
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-security (Team:Security)

n1v0lg commented 1 year ago

Oddly enough it looks like it used to fail a year or so ago in a similar way on 8.0:

https://github.com/elastic/elasticsearch/issues/83140

That issue was closed since the failure only occurred on 8.0

davidkyle commented 1 year ago

Failures on main an 8.7, mute incoming

https://gradle-enterprise.elastic.co/s/4d3s6o27awszi/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.authz.store.NativePrivilegeStoreCacheTests/testRolesCacheIsClearedWhenPrivilegesIsChanged?top-execution=1