search

jasonish / evebox

Web Based Event Viewer (GUI) for Suricata EVE Events in Elastic Search
https://evebox.org/
MIT License
418 stars 67 forks source link

Error alert query failed #180

Closed ngms17 closed 2 years ago

ngms17 commented 3 years ago

Evebox is giving me this error.

Jun 21 14:48:53 suricata-elk01 evebox[13724]: 2021-06-21 14:48:53 ERROR evebox::server::api::api: alert query failed: elasticsearch:
Jun 21 14:48:54 suricata-elk01 evebox[13724]: 2021-06-21 14:48:54 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
Jun 21 14:49:09 suricata-elk01 evebox[13724]: 2021-06-21 14:49:09 ERROR evebox::server::api::api: alert query failed: elasticsearch:
Jun 21 14:49:10 suricata-elk01 evebox[13724]: 2021-06-21 14:49:10 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
Jun 21 14:49:25 suricata-elk01 evebox[13724]: 2021-06-21 14:49:25 ERROR evebox::server::api::api: alert query failed: elasticsearch:
Jun 21 14:49:26 suricata-elk01 evebox[13724]: 2021-06-21 14:49:26 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
Jun 21 14:49:40 suricata-elk01 evebox[13724]: 2021-06-21 14:49:40 ERROR evebox::server::api::api: alert query failed: elasticsearch:
Jun 21 14:49:41 suricata-elk01 evebox[13724]: 2021-06-21 14:49:41 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
Jun 21 14:49:56 suricata-elk01 evebox[13724]: 2021-06-21 14:49:56 ERROR evebox::server::api::api: alert query failed: elasticsearch:
Jun 21 14:49:57 suricata-elk01 evebox[13724]: 2021-06-21 14:49:57 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
jasonish commented 3 years ago

Hmm.. Either Elastic is returning an empty error message or some format thats not being parsed.

Was this working before and then stopped? What version Elastic? Any error on the Elastic side? I've seen it error out there with little or no feedback to the client.

ngms17 commented 3 years ago

It was working before and then stopped. My elastic version is 7.13.1

This is es logs

[2021-06-27T00:48:52,430][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:694) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:738) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1283) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1392) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1366) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:157) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:178) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:278) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:258) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:177) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:140) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:117) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:368) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
[2021-06-27T00:49:32,614][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException:
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:89) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [65536] but was [65537]. This limit can be set by changing the [search.max_buckets] cluster level setting.
        at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:108) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregation$ReduceContext.consumeBucketsAndMaybeBreak(InternalAggregation.java:129) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:297) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:262) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceBucket(AbstractInternalTerms.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceMergeSort(AbstractInternalTerms.java:180) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:276) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:262) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceBucket(AbstractInternalTerms.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceMergeSort(AbstractInternalTerms.java:180) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:276) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.LongTerms.reduce(LongTerms.java:174) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:191) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:480) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:468) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:131) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:98) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:36) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:84) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
[2021-06-27T00:50:10,505][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:694) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:738) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1283) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1392) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1366) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:157) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:178) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:278) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:258) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:177) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:140) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:117) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:368) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
[2021-06-27T00:51:02,351][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:694) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:738) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1283) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1392) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1366) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:178) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:278) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:258) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:177) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:140) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:117) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:368) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
[2021-06-27T00:51:45,244][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException:
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:89) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [65536] but was [65537]. This limit can be set by changing the [search.max_buckets] cluster level setting.
        at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:108) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregation$ReduceContext.consumeBucketsAndMaybeBreak(InternalAggregation.java:129) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:297) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:262) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceBucket(AbstractInternalTerms.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceMergeSort(AbstractInternalTerms.java:180) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:276) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:262) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceBucket(AbstractInternalTerms.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceMergeSort(AbstractInternalTerms.java:180) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:276) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.LongTerms.reduce(LongTerms.java:174) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:191) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:480) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:468) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:131) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:98) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:36) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:84) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
[2021-06-27T00:51:46,434][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException:
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:89) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [65536] but was [65537]. This limit can be set by changing the [search.max_buckets] cluster level setting.
        at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:108) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregation$ReduceContext.consumeBucketsAndMaybeBreak(InternalAggregation.java:129) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:297) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:262) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceBucket(AbstractInternalTerms.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceMergeSort(AbstractInternalTerms.java:180) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:276) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:262) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceBucket(AbstractInternalTerms.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduceMergeSort(AbstractInternalTerms.java:180) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:276) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.LongTerms.reduce(LongTerms.java:174) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:248) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:191) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:480) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:468) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:131) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:98) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:36) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:84) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
[2021-06-27T00:52:12,477][WARN ][r.suppressed             ] [suricata-elk01] path: /filebeat-*/_search, params: {rest_total_hits_as_int=true, index=filebeat-*}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:694) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:738) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1283) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1392) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1366) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:114) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.TopHitsAggregator.buildAggregation(TopHitsAggregator.java:178) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.metrics.MetricsAggregator.buildAggregations(MetricsAggregator.java:41) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.access$900(GlobalOrdinalsStringTermsAggregator.java:54) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:742) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$StandardTermsResults.buildSubAggs(GlobalOrdinalsStringTermsAggregator.java:692) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:608) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$ResultStrategy.access$200(GlobalOrdinalsStringTermsAggregator.java:557) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.buildAggregations(GlobalOrdinalsStringTermsAggregator.java:182) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollector$2.buildAggregations(BestBucketsDeferringCollector.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForBuckets(BucketsAggregator.java:162) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.buildSubAggsForAllBuckets(BucketsAggregator.java:225) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.access$300(NumericTermsAggregator.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:278) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$StandardTermsResultStrategy.buildSubAggs(NumericTermsAggregator.java:258) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.buildAggregations(NumericTermsAggregator.java:177) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator$ResultStrategy.access$200(NumericTermsAggregator.java:140) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.bucket.terms.NumericTermsAggregator.buildAggregations(NumericTermsAggregator.java:117) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:143) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:71) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:145) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:368) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1]
        ... 6 more
jasonish commented 3 years ago

Did it start to fail shortly or right after an upgrade to 7.13? After running 7.10 or 7.11 for example?

jasonish commented 3 years ago

I don't have a sure fix for this without moving to a composite aggregation which I know I won't get to soon. So I've lowered the bucket sizes in the query which might work. This is now in my development builds at:

https://evebox.org/files/development/

jasonish commented 2 years ago

I think lowering the bucket size has shown to be a reasonable fix and this is in git master.