search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.16k stars 24.84k forks source link

Error logging during recoverable error in GeoIpDownloader #100459

Open masseyke opened 1 year ago

masseyke commented 1 year ago

The GeoIpDownloader can fail with recoverable errors while in processDatabase, but it currently does not retry and instead just gives up on the database it was processing when the error happened. A related problem is that it logs a message at error level -- this causes unnecessary concern when (for example) the cluster is shutting down. Here are the two ways I've seen it fail (in both cases the error message was something like error downloading geoip database [GeoLite2-Country.mmdb]):

org.elasticsearch.node.NodeClosedException: node closed {es-es-search-6d94b54b85-wrtbs}{YEuKOL3ETDaWQ8cmELqc_g}{UR4m3anaQGGy1fPdtfUqmA}{es-es-search-6d94b54b85-wrtbs}{10.2.8.179}{10.2.8.179:9300}{Srt}{8.11.0}{7000099-8500003}{ml.config_version=11.0.0, stateless.memory=8589934592, xpack.installed=true, k8s_node_name=ip-10-20-149-110.ec2.internal, transform.config_version=10.0.0}
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:1019)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:348)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:275)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.service.ClusterApplierService.addTimeoutListener(ClusterApplierService.java:254)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:178)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:119)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:111)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:1011)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:990)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:988)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:238)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:974)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:938)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$2.sendRequest(SecurityServerTransportInterceptor.java:270)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:449)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendRequestInner(SecurityServerTransportInterceptor.java:245)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:175)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:842)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:752)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationAction.java:957)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performRemoteAction(TransportReplicationAction.java:948)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:881)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:1014)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:341)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:171)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:119)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:111)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:1011)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:990)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1421)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:1305)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:238)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService.onConnectionClosed(TransportService.java:1301)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.ConnectionManager$DelegatingNodeConnectionListener.onConnectionClosed(ConnectionManager.java:85)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.ClusterConnectionManager.lambda$internalOpenConnection$8(ClusterConnectionManager.java:369)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:128)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:275)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:173)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.CloseableConnection.close(CloseableConnection.java:46)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TcpTransport$NodeChannels.close(TcpTransport.java:299)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:71)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.closeWhileHandlingException(IOUtils.java:169)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.ClusterConnectionManager.pendingConnectionsComplete(ClusterConnectionManager.java:351)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:117)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:69)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.ClusterConnectionManager.internalClose(ClusterConnectionManager.java:334)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.ClusterConnectionManager.close(ClusterConnectionManager.java:323)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:71)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:87)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:63)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.transport.TransportService.doStop(TransportService.java:349)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:80)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.node.Node.stopIfStarted(Node.java:1667)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.node.Node.stopIfStarted(Node.java:1661)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.node.Node.stop(Node.java:1651)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.node.Node.close(Node.java:1679)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:71)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:87)
    at org.elasticsearch.base@8.11.0/org.elasticsearch.core.IOUtils.close(IOUtils.java:63)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.bootstrap.Elasticsearch.shutdown(Elasticsearch.java:469)
    at java.base/java.lang.Thread.run(Thread.java:1583)
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of TimedRunnable{original=org.elasticsearch.action.bulk.TransportBulkAction$2/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener/org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener/org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.tasks.TaskManager$1{SafelyWrappedActionListener[listener=org.elasticsearch.action.support.ContextPreservingActionListener/org.elasticsearch.action.support.PlainActionFuture@4c7101a6]}{Task{id=1504, type='transport', action='indices:data/write/index', description='index {[.geoip_databases][GeoLite2-Country.mmdb_0_1696436679560], source[n/a, actual length: [1mb], max length: 2kb]}', parentTask=unset, startTime=1696436679629, startTimeNanos=3105367714842621}}/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda/0x0000001802232910@7319bb36/org.elasticsearch.action.bulk.TransportBulkAction$$Lambda/0x000000180265c438@212ae917/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter$$Lambda/0x0000001802232910@321ecf2e/org.elasticsearch.action.bulk.TransportBulkAction$$Lambda/0x000000180265cf58@1dbbdca2, creationTimeNanos=3105367715119660, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = es-es-search-6d94b54b85-wrtbs/write, queue capacity = 10000, task execution EWMA = 3.7ms, total task execution time = 92.5ms, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@16c3388e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 18]] (shutdown)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.EsRejectedExecutionHandler.newRejectedException(EsRejectedExecutionHandler.java:40)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:34)
    at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
    at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1376)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:72)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.bulk.TransportBulkAction.forkAndExecute(TransportBulkAction.java:261)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.bulk.TransportBulkAction.ensureClusterStateThenForkAndExecute(TransportBulkAction.java:256)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:215)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:92)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.serverless.apifiltering@8.11.0/co.elastic.elasticsearch.api.validation.PublicSettingsValidationActionFilter.apply(PublicSettingsValidationActionFilter.java:101)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:623)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:602)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:309)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:202)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:47)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:27)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.serverless.apifiltering@8.11.0/co.elastic.elasticsearch.api.validation.PublicSettingsValidationActionFilter.apply(PublicSettingsValidationActionFilter.java:101)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.api.filtering.ApiFilteringActionFilter.apply(ApiFilteringActionFilter.java:53)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:623)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:617)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:602)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:309)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:202)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:110)
    at org.elasticsearch.xcore@8.11.0/org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:164)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:154)
    at org.elasticsearch.security@8.11.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:197)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:59)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.OriginSettingClient.doExecute(OriginSettingClient.java:43)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:365)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.client.internal.support.AbstractClient.index(AbstractClient.java:394)
    at org.elasticsearch.ingest.geoip@8.11.0/org.elasticsearch.ingest.geoip.GeoIpDownloader.indexChunks(GeoIpDownloader.java:231)
    at org.elasticsearch.ingest.geoip@8.11.0/org.elasticsearch.ingest.geoip.GeoIpDownloader.processDatabase(GeoIpDownloader.java:181)
    at org.elasticsearch.ingest.geoip@8.11.0/org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143)
    at org.elasticsearch.ingest.geoip@8.11.0/org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:279)
    at org.elasticsearch.server@8.11.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-data-management (Team:Data Management)