[Self-Managed][Fleet]: Data under data streams tab is not completely visible and datasets keep on increasing and decreasing inconsistently.

[amolnater-qasource]

Kibana version: 8.5 BC3 Kibana self-managed environment

License: Platinum

Host OS: Windows

Build details: VERSION: 8.5 BC3 Kibana self-managed environment BUILD: 56952 COMMIT: 565ddb404984c516dab6ce46fd0c82c42d0c9dc3

Preconditions:

1. 8.5 BC3 Kibana self-managed environment should be available.
2. Fleet Server and multiple agent should be installed with different namespace. UPDATE: Secondary agent should be installed on self-managed environment.

Steps to reproduce:

1. Navigate to Data Streams tab.
2. Observe pagination is only 1 and datasets are continuously changing on hard refresh.
3. Apply any filter and observe same behaviour.

What's working fine: This issue is not observed on cloud-production environment.

Expected Result: Data under data streams tab should be consistently available and pagination should be working fine.

Screen Recording:

https://user-images.githubusercontent.com/77374876/195336617-d972e0f9-22be-475a-bbc6-c3bb678a2637.mp4

https://user-images.githubusercontent.com/77374876/195347782-d7ac30d4-caa1-4034-b505-19029a937af0.mp4

Logs:

elastic-agent-diagnostics-2022-10-12T11-01-11Z-00.zip

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@karanbirsingh-qasource Please review

[ghost]

Secondary review for this ticket is Done

[ghost]

Bug Conversion

• Test-Case not required as this particular checkpoint is already covered in exploratory testing.

  Thanks!

[amolnater-qasource]

Hi @jlind23 We have revalidated this issue on latest 8.6 BC6 kibana self-managed environment and found it still reproducible.

Observations:

• Data under data streams tab is not completely visible and datasets keep on increasing and decreasing inconsistently.

Build details: 8.6 BC6 Self-managed environment BUILD: 58740 COMMIT: f329a77595950244361736dff7208a810299fd69

Screen Recording:

https://user-images.githubusercontent.com/77374876/207020168-a06a3648-38be-4ddf-824e-a16d244d2715.mp4

Please let us know if anything else is required from our end. Thanks

[amolnater-qasource]

Hi Team, We have revalidated this issue on latest 8.7.0 BC3 self-managed environment and found it still reproducible.

Observations:

• When secondary agent is installed, data under data streams tab is not completely visible and datasets keep on increasing and decreasing inconsistently.

Build details: BUILD: 60783 COMMIT: 4ed6814a5dd5532a30bde9b34354a949d9166ae9

Screen Recording:

https://user-images.githubusercontent.com/77374876/220251217-b50528a0-86fb-46d5-8a3f-1bfdcdc08548.mp4

Please let us know if anything else is required from our end. Thanks

[amolnater-qasource]

Hi @jlind23

While testing on 8.9 Snapshot self-managed environment, we have found this issue still reproducible.

Observations:

• When secondary agent is installed, data under data streams tab is not completely visible and datasets keep on increasing and decreasing inconsistently.

Screen Recording:

https://github.com/elastic/kibana/assets/77374876/5ddf1384-ca97-4397-a130-2ab80d6fe121

Build details: BUILD: 64236 COMMIT: c191104ba1b2b12d693e8e9e2b0f34e092da1e9a

Please let us know if anything else is required from our end.

Thanks!

[criamico]

@amolnater-qasource This is a very old ticket, I tried to reproduce on 8.12.0 and I'm not able to. Could you please test it again?

I suspect that this got solved back in 8.9.0 by https://github.com/elastic/kibana/pull/159253, we just need to check if it's still occurring.

[amolnater-qasource]

Hi @criamico

We have revalidated this issue on latest 8.13.0-SNAPSHOT self-managed environment and found the issue still reproducible.

Steps to reproduce:

1. Setup self-managed 8.13.0-SNAPSHOT environment.
2. Installed a Fleet Server and an additional agent on self-managed environment.
3. Restarted the elasticsearch and kibana instances.[Additional Step]
4. Navigate to Fleet>Data Streams.
5. Refresh using the button available.
6. Observe data under data streams tab is not completely visible and datasets keep on increasing and decreasing inconsistently.

Screen Recording:

https://github.com/elastic/kibana/assets/77374876/95671cab-c08e-400b-8bb0-1dcbdf05fb3a

Build details: VERSION: 8.13.0-SNAPSHOT self-managed BUILD: 71681 COMMIT: 1b797a52e6cb74ea649ea85ed8723a9bc8baec6e

Please let us know if anything else is required from our end. Thanks!

[criamico]

@amolnater-qasource Thanks for testing again! I've been trying to reproduce locally with no success. That's what I've done:

• Enrolled a fleet server
• Enrolled a dockerized agent (default namespace)
• Enrolled a second dockerized agent (testnamespace namespace)
• In the data streams tab, I tried to refresh several times, also with/without filters. In all these case I always see the same amount of data streams, the only thing changing is the order (which is correct since we order by "last activity"). But I couldn't see the number of data streams changing as in the videos above. Is it possible that I'm missing something in the above conditions?

[amolnater-qasource]

@criamico Thank you for the update.

Could you please stop elasticsearch and kibana instances and rerun the same, if that reproduces the issue?

Further for more details we are consistently able to reproduce this issue on AWS-Windows 2022 server, localhost running on Mozilla Firefox browser.

Thanks!

[criamico]

@amolnater-qasource I've tried restarting both Kibana and ES and I keep seeing the same number of data streams (video below).

https://github.com/elastic/kibana/assets/16084106/db12dc39-b503-464b-80d8-454c9bb9862b

One possible explanation could point to connection issues with ES, causing the underlying queries to get only a subsection of data streams. Also, in the code we are filtering out results in several places and especially the following block could explain the different results (filter out data streams that are missing dataset/namespace/type/package fields):

https://github.com/elastic/kibana/blob/e1748f9368c0436d605f1f887f18d30c6feb6679/x-pack/plugins/fleet/server/routes/data_streams/handlers.ts#L203-L211

I'll try to find a different way to reproduce it. So far I've tried on self managed and on cloud but I don't see this issue in any of them.

[criamico]

I spoke with Amol and was able to log in into a VM that has this issue. I looked at the data streams indices in the Index Management view but I couldn't find anything that could explain why some of them are randomly not returned in the response of /api/fleet/data_streams. The only thing is that "health" is yellow, but most of those indices are returned anyway.

So I'm thinking that the reason why sometimes we see missing indices is this filter: https://github.com/elastic/kibana/blob/e1748f9368c0436d605f1f887f18d30c6feb6679/x-pack/plugins/fleet/server/routes/data_streams/handlers.ts#L203-L211

Where the data comes from getDataStreamsQueryMetadata and specifically those fields are added here:

https://github.com/elastic/kibana/blob/94f30575409c43e009af9478744c8e00b6abe9ed/x-pack/plugins/fleet/server/routes/data_streams/get_data_streams_query_metadata.ts#L89-L91

I'm not sure why the content of those fields would change so frequently and I couldn't find an obvious reason for that, but at least it explains why the length of the response can be different at every reload. It is kinda expected because of that filtering, plus we don't really test that the response has to be the same every time, however I'm not sure if this is expected behavior or not.

[criamico]

@nchaulet suggested that we are doing 50 concurrent requests here and some of them could be failing. We'd need to check ES proxy logs for that, but currently I haven't been able to reproduce it on an environment where we have access to logs.

[kpollich]

Thanks for diving in a bit here, @criamico - a few things that come to mind:

1. Is there anything unique about @amolnater-qasource's environment that we can replicate? Could we install the same integrations, number of agents, etc perhaps? Wondering if we need to get some live data being ingested as well.
2. @amolnater-qasource - Is it possible to get Elasticsearch logs from a time where you are experiencing this issue in this environment? We would like to see if there are errors when fetching data stream stats
3. Could we add some more granular logging around the existing pMap in the meantime to see if we are in fact getting errors back from Elasticsearch? We could also add retry logic.

About the pMap call and error handling. By default, pMap calls get stopOnError: true (https://www.npmjs.com/package/p-map#stoponerror), meaning the call will terminate on the first error and return that error. This is in line with the behavior of the native Promise.all function. So, if we were seeing even one request fail I would expect this call to bomb out once that error was thrown.

Could that behavior explain the divergence in data that we're seeing here? If we passed stopOnError: false (mimicking the behavior of the native Promise.allSettled function), we'd allow all the requests to finish and get an aggregate error array once the last request is finished. Maybe changing the approach there could prove helpful in tracking down the root cause of this?

[criamico]

Is there anything unique about @amolnater-qasource's environment that we can replicate? Could we install the same integrations, number of agents, etc perhaps? Wondering if we need to get some live data being ingested as well.

That is a self managed environment on a S3 bucket, I'm not sure what could be causing this issue. I tried several things:

• installing the same integrations (system and apm) using different namespaces
• used both local environment and on cloud
• Had healthy and unhealthy agents In all these cases I always got a consistent number of data streams. Could it be something different, for instance connection issues with ES?

[amolnater-qasource]

Hi @kpollich

Is there anything unique about @amolnater-qasource's environment that we can replicate? Could we install the same integrations, number of agents, etc perhaps? Wondering if we need to get some live data being ingested as well.

We have setup a fresh 8.13.0- BC1 self-managed environment and found it reproducible there too with the steps below:

1. Setup self-managed 8.13.0-BC1 environment.
2. Install a fleet server with policy having System and Fleet Server integration.
3. Till now no issue is observed under Data streams tab.
4. Install another agent using different policy with --insecure flag, no integration other than system integration is added. Restart the elasticsearch and kibana instances.[Issue reproduced without this step.]
5. Navigate to Fleet>Data Streams.
6. Refresh using the button available.
7. Observe data under data streams tab is not completely visible and datasets keep on increasing and decreasing inconsistently.

Fleet-Server: AWS-Windows 2022 Server

Second Agent: SLES15

@amolnater-qasource - Is it possible to get Elasticsearch logs from a time where you are experiencing this issue in this environment? We would like to see if there are errors when fetching data stream stats

We have copied below logs from elasticsearch>logs folder. elasticsearch-logs.zip

Please let us know if that fresh instance login details are required.

Thanks!

[kpollich]

Seeing some PIT query errors in the ES server logs, e.g.

{"@timestamp":"2024-02-22T06:09:08.286Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.310Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.322Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {expand_wildcards=open,hidden, index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.338Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.346Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {expand_wildcards=open,hidden, index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.362Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.370Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {expand_wildcards=open,hidden, index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}
{"@timestamp":"2024-02-22T06:09:08.382Z", "log.level": "WARN", "message":"path: /.alerts-security.alerts*/_pit, params: {expand_wildcards=open,hidden, index=.alerts-security.alerts*, keep_alive=5m}, status: 503", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[EC2AMAZ-4H2FTRN][transport_worker][T#6]","log.logger":"rest.suppressed","trace.id":"55eb286eaf21a5ef360f4ca3bfbaa503","elasticsearch.cluster.uuid":"bLszSuGDTrSXOFtKrigAKw","elasticsearch.node.id":"Az35XEKuT4mP4LuIAu9pNg","elasticsearch.node.name":"EC2AMAZ-4H2FTRN","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"","error.stack_trace":"Failed to execute phase [indices:data/read/open_point_in_time], \r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:459)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1166)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:914)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$8(TransportSearchAction.java:342)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:455)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:165)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:634)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:632)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:628)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:613)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:511)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1027)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:993)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:402)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:189)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:165)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1073)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:382)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:504)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:436)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:323)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:152)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)\r\n\tat java.base/java.lang.Iterable.forEach(Iterable.java:75)\r\n\tat org.elasticsearch.xcore@8.13.0/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:148)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:339)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:161)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:157)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:356)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$1(RestOpenPointInTimeAction.java:64)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:107)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)\r\n\tat org.elasticsearch.security@8.13.0/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat io.netty.codec@4.1.94.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137)\r\n\tat org.elasticsearch.transport.netty4@8.13.0/org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\r\n\tat io.netty.transport@4.1.94.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\r\n\tat io.netty.common@4.1.94.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\r\nCaused by: Failed to execute phase [indices:data/read/open_point_in_time], Search rejected due to missing shards [[.internal.alerts-security.alerts-default-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:233)\r\n\tat org.elasticsearch.server@8.13.0/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454)\r\n\t... 135 more\r\n"}

Not sure if these are related to the data streams API or not, but that seems to be the only error in the logs that I can find.

[criamico]

I also looked at the attached logs and couldn't find any errors related to data streams. There are related logs of type:

[2024-02-22T06:17:09,588][INFO ][o.e.c.m.MetadataCreateIndexService] [EC2AMAZ-4H2FTRN] [.ds-metrics-elastic_agent.fleet_server-default-2024.02.22-000001] creating index, cause [initialize_data_stream], templates [metrics-elastic_agent.fleet_server], shards [1]/[1]
[2024-02-22T06:17:09,588][INFO ][o.e.c.m.MetadataCreateDataStreamService] [EC2AMAZ-4H2FTRN] adding data stream [metrics-elastic_agent.fleet_server-default] with write index [.ds-metrics-elastic_agent.fleet_server-default-2024.02.22-000001], backing indices [], and aliases []