elasticmachine
commented
3 years ago Pinging @elastic/es-security (Team:Security)
Open albertzaharovits opened 3 years ago
elasticmachine
commented
3 years ago Pinging @elastic/es-security (Team:Security)
albertzaharovits
commented
3 years ago Investigation notes:
IndicesPermission#authorize to improve performance when FLS/DLS is not enabled.Role level, for a pair of action name and indices; also we should exploit the incremental nature of the IndicesPermission#authorize check wrt to the index set (ie if we know the result of authorize(action:x, [a]) and authorize(action:x, [b]) we don't need to recompute authorize(action:x, [a, b])).
jcsorvasi
commented
3 years ago Hey @albertzaharovits,
I think we're hitting exactly this issue in a cluster with ~2500 indices. Elasticsearch running on Kubernetes, orchestrated by ECK is exhibiting readiness probe failures that we think could be tracked back to this. A hot_threads output shows the following:
$ curl -s -k "https://elastic:PW@localhost:9200/_nodes/logging-prod-es-ingest-jmx-a-0/hot_threads" | grep "cpu usage by thread" -A 3
100.1% (500.6ms out of 500ms) cpu usage by thread 'elasticsearch[logging-prod-es-ingest-jmx-a-0][transport_worker][T#8]'
10/10 snapshots sharing following 304 elements
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:286)
--
100.1% (500.6ms out of 500ms) cpu usage by thread 'elasticsearch[logging-prod-es-ingest-jmx-a-0][transport_worker][T#3]'
4/10 snapshots sharing following 305 elements
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:536)
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367)
--
100.1% (500.5ms out of 500ms) cpu usage by thread 'elasticsearch[logging-prod-es-ingest-jmx-a-0][transport_worker][T#2]'
10/10 snapshots sharing following 304 elements
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:286)Much more detail can be found in the following forum discussion: https://discuss.elastic.co/t/elasticsearch-readiness-probe-failures/270581
(edit for clarification: the issue seems to significantly limit indexing throughput in this setup) This kind of makes x-pack security unusable for large clusters (our current v5 production cluster is several times larger than this, we're looking at upgrading now). Do you know if there is traction on this (and the related #68004)?
Thank you.
Cheers,
János
albertzaharovits
commented
3 years ago Hi @jcsorvasi!
I appreciate your input so far. We're currently investigating, but not committing for a fix just yet.
I understand there are ~2500 indices in the cluster that you're ingesting into. I have a couple of follow-up question to that:
If possible, can you create a JFR dump for the node, and share it with us?
You need to restart the node with the following in config/jvm.options:
-XX:+UnlockDiagnosticVMOptions
-XX:+DebugNonSafepoints
-XX:FlightRecorderOptions=memorysize=98M
-XX:StartFlightRecording=disk=true,name=elasticsearch-continuous,maxsize=98M
-XX:+StartAttachListenerand then issue:
$JAVA_HOME/bin/jcmd $ES_PID JFR.dumpwhen the ingest node is hiccuping. In addition to the stacktraces it would provide us with information on the memory load, to help evaluate the caching opportunities.
I am thankful for whatever details you can provide to help us!
Also, from the forum link https://discuss.elastic.co/t/elasticsearch-readiness-probe-failures/270581 it looks like you've got some CPU capacity left. In this case I think you could experiment with increasing the number of transport threads, so that, even if inefficient, authz can saturate more of the available cpu capacity, increasing throughput, ie, try setting on the ingest node:
transport.netty.worker_count: 32Hey @albertzaharovits,
Let me answer some of your questions quickly now, gathering Flight Recorder data is going to take some time as we've destroyed the cluster already (note on that in the forum post in my latest comment). I'll hopefully have an opportunity to rebuild it sometime next week and get you the data you need, as well as do some comparison between identical clusters receiving the same data, one with xpack security turned on and one with it off.
Are these datastreams, aliases or regular indices? Same question about the ones that you're ingesting into.
In short: aliases.
Our index setup is basically per-app weekly indices with daily aliases (from before datastreams were a thing). For example:
| Alias | Index |
|---|---|
logstash-app1-2021.04.16logstash-app1-2021.04.17logstash-app1-2021.04.18 |
logstash-app1-2021.15-000001 |
logstash-app1-2021.04.19logstash-app1-2021.04.20logstash-app1-2021.04.21 |
logstash-app1-2021.16-000001 |
Indices and aliases are pre-created for the future by a management tool. Just to give you some numbers, since we rebuilt the (test) cluster on Tuesday and started ingesting production data into it, the cluster now has 1116 indices, 4852 aliases and 3846 active shards. Our current live (v5) production cluster has a rolling retention of 32 days, with 7646 indices, 43126 aliases and 30244 active shards.
The indexer (logstash with the elasticsearch output) is configured with the following parameter:
index => "logstash-%{[@metadata][_appname]}-%{+YYYY.MM.dd}"This way (almost) all index requests are sent to their respective alias for the current day and so each bulk request has the potential to include up 6000 index requests, possibly touching hundreds of indices or more.
do the requests, in general, access most of the indices or just a couple?
I would say most of the indices. As I mentioned, there are per-app indices and how much an app logs varies wildly. Some of the chattiest apps sometimes produce tens of thousands of messages per second, some only a handful for the entire day.
do the roles assigned to the user use document or field level security?
We didn't go far into using all the security features, the setup is basically what ECK sets up. All apps around ES (logstash, kibana, etc.) use the built-in elastic (super)user. We haven't configured any document or field level security
can you share an approximate for the number of users, as well as for the number of roles that the user has assigned?
Vanilla ECK setup, no extra users or roles created by us.
can you please estimate the rate of change for indices? How often are new indices created/deleted?
Once a day a management tool runs that pre-creates aliases (per app) for the future, ~1300 aliases created. Once a week, this same process hits a new week and pre-creates the backing weekly indices, so ~1300 indices created on top of the ~1300 aliases once a week. Once a day, curator runs to remove indices that contain data older than 32 days. This normally produces little churn, but once a week removes a whole week's worth of data, so ~1300 indices. A separate process continuously monitors the incoming data streams per app and if it detects a data spike that it deems the current shard configuration for the index wouldn't be able to handle, it executes a rollover (with a higher number of shards). This happens 0-17 times per day, on average 3 times, so ~3 indices created per day.
Thanks for the tip on increasing the transport thread count, I'll try it out next week!
ilavajuthy
commented
2 years ago Hi @albertzaharovits : We are facing exactly similar performance issue with authorization. We have around 3K indices and 20K aliases. After upgrading to 7.9.0 with XPACK authentication, all the ingest and search operations have slowed down drastically. Our ingest performance use to be around 20ms, now its in 500ms.
All the hot_threads are showing transport_worker with loadAuthorizedIndices stack trace.
Also we are noticing CPU is not fully utilized in each POD, however the CPU % from _cat/nodes is showing 100% usage.
We have created a user and assigned superuser role.
Is there any recommended tuning to improve the performance?
Is there a way to disable index authorization?
orinciog
commented
2 years ago Hi!
We also have a large cluster impacted by this issue. Our setup is 5K+ indices, 25K+ aliases, write rate 60K/sec, 50 machines with 24 ingesting nodes. We write in about 100 indices.
We upgraded from ES 6.2.3 with search guard to ES 7.9.3 with xpack. Ingesting times for our setup are:
Our stacks from hot_threads are similar to those provided in the first topic. Also are similar in search guard case.
Also, the load of masters is quite different:
Also, the interval of full cluster restart is different:
albertzaharovits
commented
2 years ago @ilavajuthy @orinciog Thank you for the new imput on this one!
TLDR I strongly recommend you upgrade to 7.16, which just came out . Or at least I would be grateful if you could run the same benchmarks on it. That release includes a couple of performance-related changes, including, but not limited: Filter original indices in shard level request by jimczi · Pull Request #78508 · elastic/elasticsearch Superuser fastpath for indexAccessControl by ywangd · Pull Request #78498 · elastic/elasticsearch Skip loading authorized indices if requests do not need them by ywangd · Pull Request #78321 · elastic/elasticsearch Avoid redundant authz indices contains check for wildcard expansion by albertzaharovits · Pull Request #76540 · elastic/elasticsearch
There are many changes between the 6.2 and 7.9 releases, but honestly we didn't benchmark comparatively. I suspect, though, that the performance issues might not be directly caused by the Security code changes.
At the moment we are comfortable with the state of this in 7.16, and we're not planning any changes soon. But, if you have a comparative benchmark with a 6.x release please do let me know, I promise I'll investigate.
Thank you again for taking the time to report it.
orinciog
commented
2 years ago Hi! @albertzaharovits Thank you for your input on this problem.
Right now, for us it's a lit bit tricky to update to 7.16 (We have a series of Kibana legacy plugins and we are stuck with Kibana 7.9.3 - the latest version of Kibana that supports legacy plugins)
I want to ask you if you compared the performance of es 7.x and 7.16 in terms of large clusters (high number of indices, high ingest rate) and if you have some comparatives between them (with xpack and without xpack).
From our numbers, in 7.9.3, with xpack enabled the performance is about 10x worst than without xpack (all others factors remain the same - ingesting rate, number of machines etc).
If you have such comparatives and the results of them are ok, than we could try to migrate to latest version.
Thank you again for your effort and input,
ilavajuthy
commented
2 years ago Hi @albertzaharovits : Thanks for the details that 7.16 have fixes to improve performance. Moving to 7.16 will take sometime for us. In the meantime is there any option to disable authorization and have only authentication for super user roles?
albertzaharovits
commented
2 years ago Hi @orinciog! We unfortunately do not have benchmarks for large clusters. Our benchmarks, that we use to detect regression, use clusters of three nodes and only a couple of indices, for eg https://elasticsearch-benchmarks.elastic.co/#tracks/nyc-taxis/nightly/default/1y . The truth is that we've been blind to clusters with many names (indices, aliases, data streams), and some eggregious inefficiencies have crept in the authorization code. We'll be adding benchmarks for many indices.
I can't reliabily help you out with benchmarking, I'm sorry. Realistically, I would only have bandwidth to set-up and run a rally test bench if I could tie it up to some further performance improvements, but we're not planning any of that for now (we're relatively content with the changes we got in 7.16).
If you go about benchmarking on your own, you can start from this recent blog post https://www.elastic.co/blog/seven-tips-for-better-elasticsearch-benchmarks , and please do let us know about the test scenario (total number of indices/aliases/data streams, the number of indices/aliases/data streams that the user(s) are authorized for, the number of cluster nodes, whether you're benchmarking ingestion or searching and if searching, how many indices are searched in a single request (search requests can contain wildcards)).
Hi @ilavajuthy Generally speaking some performance issues can be avoided if users only have access to a relatively small subset of indices (indeed it is generally a good idea to avoid users with broad capabilities). If you find it helpful, you could employ a run-as scheme, where the user with broad capabilities, instead of directly indexing/querying, uses the run-as header as a way to restrict the visibility for the operation.
Another option would be to implement a plugin with a custom authorization engine, see:
https://github.com/elastic/elasticsearch/tree/master/plugins/examples/security-authorization-engine
https://github.com/elastic/elasticsearch/tree/master/plugins/examples/security-authorization-engine/src/main/java/org/elasticsearch/example
This approach is also uncomfortable because plugins have to be recompiled and redeployed on every upgrade.
In addition it is fraught because the authorization methods are not ideally abstracted, and there are undocumented subtlelties when implementing the AuthorizationEngine interface.
But it is technically doable and there is an example.
Mannoj87
commented
1 year ago @albertzaharovits run_as doesn't work. We tried via run_as and still could see RBAC and index.Permission related ones. Any other alternative other than upgrade?
nitishgoyal13
commented
1 year ago We run a massive 300+ servers cluster and we started seeing drop in throughput as our number on indices increased. We use daily indices approach for our Elasticsearch cluster. We went live with 7.10.2 as we were seeing similar performance when we ran the perf tests for few days. We didn't hit the higher indices issue during our perf test days since we ran the perf only for 7 days.
After, we went live in production, after 15 days, as the number of indices increased, we started seeing slowness in our ingestion pipelines
Upon further investigation, we figured out RBAC Engine is the one causing slowness for the overall cluster.
We did a rolling upgrade from 7.10.2 to 7.17.8 version and we are seeing a massive 2x-3x throughput on the same hardware and with same workload
7.10.2 Version - For a 300 node cluster, we were earlier not able to go beyond 450k indexing operations per second. 7.17.8 Version - We are able to do around 1 million indexing operations per second
The
RBACEngine#resolveAuthorizedIndicesForRolecomputes a list of all the indices that a role grants access to, given the request action. This list is subsequently used during the evaluation of the request's index name expression.The issue is that the list can be large. Worse, the expensive, in both time and memory, list creation (eg the repeated resizings) might take place on
transport_workerthreads that have to multiplex (select) reads between multiple TCP connections (channels, in netty/nio terminology), effectively delaying responses on the other connections (which might lead to cluster instability).A list is wasteful because the request's wildcard will generally only match a small subset of the full list, which is the only data that needs to be stored in the re-written request. Ideally, the "authorized indices" full list should instead be an
Iterablegenerator, which doesn't store all the names in memory. The generator refactoring, can also be improved by another refactoring, where we traverse the authorized indices only once.Here's a very long list of very long stack traces from a hot threads output, that I've based my above analysis on:
``` 100.5% (502.3ms out of 500ms) cpu usage by thread 'elasticsearch[cf75d9f511c336298a744fdd552abb85cf75d9f][transport_worker][T#1]' 10/10 snapshots sharing following 330 elements org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5139/0x000000080182c428.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5140/0x000000080182c650.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5143/0x000000080182cce8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4818/0x0000000801616018.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4820/0x0000000801616478.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4921/0x0000000801795be8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4778/0x0000000801615bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4779/0x0000000801615df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shardExecute(TransportBroadcastReplicationAction.java:118) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:111) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:52) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) app//org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4918/0x0000000801795550.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4923/0x0000000801796048.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5158/0x000000080182e2c8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4748/0x00000008017691d8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:75) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5156/0x000000080182de68.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4748/0x00000008017691d8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5156/0x000000080182de68.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4748/0x00000008017691d8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:106) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5156/0x000000080182de68.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4748/0x00000008017691d8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5156/0x000000080182de68.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4748/0x00000008017691d8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:366) org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:343) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:285) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5141/0x000000080182c878.accept(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:641) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:541) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:321) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5145/0x000000080182d148.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5147/0x000000080182d5a8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:579) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:273) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5149/0x000000080182da08.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5147/0x000000080182d5a8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5139/0x000000080182c428.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5140/0x000000080182c650.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5143/0x000000080182cce8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4818/0x0000000801616018.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4820/0x0000000801616478.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4921/0x0000000801795be8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4778/0x0000000801615bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4779/0x0000000801615df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) app//org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) app//org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:409) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1274) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.refresh(AbstractClient.java:1585) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction.lambda$prepareRequest$0(RestRefreshAction.java:60) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction$$Lambda$5558/0x00000008018d4f50.accept(Unknown Source) app//org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:76) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5181/0x0000000801841200.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:82) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$5197/0x0000000801843570.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:92) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:77) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:70) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$4978/0x0000000801796e20.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:680) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5171/0x000000080182fcb8.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:695) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:669) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:616) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:493) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5067/0x000000080181b2b0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:459) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5092/0x000000080181d5c8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:146) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5168/0x000000080182eda0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:197) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:138) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5115/0x0000000801828b00.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:133) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:448) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5060/0x000000080181a308.accept(Unknown Source) org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:503) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5030/0x00000008018160c0.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:415) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5051/0x0000000801818b08.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:425) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:366) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5014/0x0000000801813b88.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:342) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4995/0x0000000801811438.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:388) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4778/0x0000000801615bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4779/0x0000000801615df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:63) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) app//org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) app//org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:318) app//org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) app//org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base@15/java.lang.Thread.run(Thread.java:832) 100.4% (502.2ms out of 500ms) cpu usage by thread 'elasticsearch[42dc5d67f7823cd2bfaad53ab1985e1d42dc5d6][transport_worker][T#5]' 4/10 snapshots sharing following 333 elements java.base@15/java.util.HashMap.put(HashMap.java:612) java.base@15/java.util.HashSet.add(HashSet.java:221) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:522) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5197/0x0000000801826710.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5198/0x0000000801826938.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5201/0x0000000801826fd0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4866/0x0000000801607af8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4868/0x00000008017d0040.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4862/0x0000000801607238.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shardExecute(TransportBroadcastReplicationAction.java:118) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:111) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:52) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) app//org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4859/0x0000000801606ba0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4864/0x0000000801607698.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5216/0x00000008018386d8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:75) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:106) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:366) org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:343) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:285) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5199/0x0000000801826b60.accept(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:641) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:541) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:321) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5203/0x0000000801827430.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5205/0x0000000801827890.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:579) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:273) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5207/0x0000000801827cf0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5205/0x0000000801827890.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5197/0x0000000801826710.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5198/0x0000000801826938.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5201/0x0000000801826fd0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4866/0x0000000801607af8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4868/0x00000008017d0040.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4862/0x0000000801607238.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) app//org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) app//org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:409) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1274) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.refresh(AbstractClient.java:1585) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction.lambda$prepareRequest$0(RestRefreshAction.java:60) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction$$Lambda$6003/0x000000080192ed78.accept(Unknown Source) app//org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:76) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5235/0x000000080183abe0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:82) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$5259/0x000000080183e090.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:92) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:77) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:70) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5087/0x00000008017d7498.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:680) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5234/0x000000080183a9b8.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:695) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:669) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:616) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:493) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5137/0x000000080181e560.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:459) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5152/0x000000080181f6e0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:146) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5225/0x0000000801838f88.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:197) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:138) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5176/0x0000000801823490.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:133) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:448) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5133/0x000000080181dc60.accept(Unknown Source) org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:503) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5130/0x000000080181d5d8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:415) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5132/0x000000080181da38.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:425) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:366) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5116/0x000000080181b4f0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:342) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5105/0x0000000801819878.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:388) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:63) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) app//org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) app//org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:318) app//org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) app//org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base@15/java.lang.Thread.run(Thread.java:832) 3/10 snapshots sharing following 333 elements org.elasticsearch.xpack.core.security.authz.permission.IndicesPermission$Group.lambda$buildIndexMatcherPredicateForAction$0(IndicesPermission.java:453) org.elasticsearch.xpack.core.security.authz.permission.IndicesPermission$Group$$Lambda$5211/0x000000080180e168.test(Unknown Source) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:520) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5197/0x0000000801826710.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5198/0x0000000801826938.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5201/0x0000000801826fd0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4866/0x0000000801607af8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4868/0x00000008017d0040.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4862/0x0000000801607238.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shardExecute(TransportBroadcastReplicationAction.java:118) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:111) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:52) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) app//org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4859/0x0000000801606ba0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4864/0x0000000801607698.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5216/0x00000008018386d8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:75) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:106) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:366) org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:343) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:285) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5199/0x0000000801826b60.accept(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:641) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:541) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:321) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5203/0x0000000801827430.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5205/0x0000000801827890.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:579) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:273) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5207/0x0000000801827cf0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5205/0x0000000801827890.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5197/0x0000000801826710.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5198/0x0000000801826938.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5201/0x0000000801826fd0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4866/0x0000000801607af8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4868/0x00000008017d0040.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4862/0x0000000801607238.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) app//org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) app//org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:409) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1274) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.refresh(AbstractClient.java:1585) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction.lambda$prepareRequest$0(RestRefreshAction.java:60) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction$$Lambda$6003/0x000000080192ed78.accept(Unknown Source) app//org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:76) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5235/0x000000080183abe0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:82) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$5259/0x000000080183e090.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:92) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:77) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:70) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5087/0x00000008017d7498.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:680) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5234/0x000000080183a9b8.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:695) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:669) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:616) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:493) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5137/0x000000080181e560.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:459) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5152/0x000000080181f6e0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:146) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5225/0x0000000801838f88.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:197) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:138) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5176/0x0000000801823490.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:133) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:448) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5133/0x000000080181dc60.accept(Unknown Source) org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:503) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5130/0x000000080181d5d8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:415) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5132/0x000000080181da38.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:425) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:366) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5116/0x000000080181b4f0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:342) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5105/0x0000000801819878.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:388) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:63) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) app//org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) app//org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:318) app//org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) app//org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base@15/java.lang.Thread.run(Thread.java:832) 2/10 snapshots sharing following 331 elements org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizedIndicesFromRole(RBACEngine.java:518) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5197/0x0000000801826710.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5198/0x0000000801826938.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5201/0x0000000801826fd0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4866/0x0000000801607af8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4868/0x00000008017d0040.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4862/0x0000000801607238.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shardExecute(TransportBroadcastReplicationAction.java:118) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:111) app//org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:52) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) app//org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4859/0x0000000801606ba0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4864/0x0000000801607698.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5216/0x00000008018386d8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:75) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:106) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5214/0x0000000801838278.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$4740/0x0000000801759ad8.accept(Unknown Source) java.base@15/java.util.ArrayList.forEach(ArrayList.java:1511) app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:366) org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:343) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:285) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5199/0x0000000801826b60.accept(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:641) org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:541) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:321) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5203/0x0000000801827430.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5205/0x0000000801827890.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:579) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:273) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5207/0x0000000801827cf0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:678) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$5205/0x0000000801827890.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:352) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:269) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5197/0x0000000801826710.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:272) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$5198/0x0000000801826938.getAsync(Unknown Source) org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:676) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:313) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$5201/0x0000000801826fd0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:337) org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:310) org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:283) org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:248) org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:212) org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4866/0x0000000801607af8.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:123) org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4868/0x00000008017d0040.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:274) org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:129) org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:117) org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:214) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda$4862/0x0000000801607238.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) app//org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) app//org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) app//org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) app//org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) app//org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:409) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1274) app//org.elasticsearch.client.support.AbstractClient$IndicesAdmin.refresh(AbstractClient.java:1585) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction.lambda$prepareRequest$0(RestRefreshAction.java:60) app//org.elasticsearch.rest.action.admin.indices.RestRefreshAction$$Lambda$6003/0x000000080192ed78.accept(Unknown Source) app//org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:76) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5235/0x000000080183abe0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:82) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator$$Lambda$5259/0x000000080183e090.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:92) org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:77) org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:70) org.elasticsearch.xpack.security.rest.SecurityRestFilter$$Lambda$5087/0x00000008017d7498.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:680) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5234/0x000000080183a9b8.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:695) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:669) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:616) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:493) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5137/0x000000080181e560.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:459) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5152/0x000000080181f6e0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:146) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5225/0x0000000801838f88.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:197) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:138) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$5176/0x0000000801823490.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) app//org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:68) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:133) org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:448) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5133/0x000000080181dc60.accept(Unknown Source) org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:503) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5130/0x000000080181d5d8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:415) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5132/0x000000080181da38.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:425) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:366) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5116/0x000000080181b4f0.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:342) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$5105/0x0000000801819878.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:388) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4765/0x0000000801605bb8.accept(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$4766/0x0000000801605df0.run(Unknown Source) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:63) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:236) app//org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:318) app//org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176) app//org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:318) app//org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) app//org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:308) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) java.base@15/java.lang.Thread.run(Thread.java:832) unique snapshot java.base@15/java.util.HashMap.keysToArray(HashMap.java:951) java.base@15/java.util.HashSet.toArray(HashSet.java:367) java.base@15/java.util.ArrayList.