sothawo
commented
7 months ago You obviously have a problem on your Elasticsearch server and therefore the connection from the client times out. This is nothing that Spring Data Elasticsearch could fix.
Besides that, Spring Data Elasticsearch 4.4 is out of maintenance for almost a year now.
This is the spring-data-elasticsearch log： org.springframework.dao.DataAccessResourceFailureException: 30,000 milliseconds timeout on connection http-outgoing-5603 [ACTIVE]; nested exception is java.lang.RuntimeException: 30,000 milliseconds timeout on connection http-outgoing-5603 [ACTIVE] at org.springframework.data.elasticsearch.core.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:94) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:601) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12] at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:584) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12] at org.springframework.data.elasticsearch.core.RestIndexTemplate.doExists(RestIndexTemplate.java:106) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12] at org.springframework.data.elasticsearch.core.AbstractIndexTemplate.exists(AbstractIndexTemplate.java:130) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12] at com.proinnova.system.job.service.impl.ESJobServiceImpl.checkIndex(ESJobServiceImpl.java:205) ~[classes!/:2.7.12] at com.proinnova.system.job.service.impl.JobServiceImpl.jobCreate(JobServiceImpl.java:166) ~[classes!/:2.7.12] at com.proinnova.system.job.service.impl.JobServiceImpl$$FastClassBySpringCGLIB$$81d361fa.invoke() ~[classes!/:2.7.12]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.27.jar!/:5.3.27]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.27.jar!/:5.3.27]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.27.jar!/:5.3.27]
at com.proinnova.system.job.service.impl.JobServiceImpl$$EnhancerBySpringCGLIB$$891d2d84.jobCreate() ~[classes!/:2.7.12]
at com.proinnova.system.job.rest.JobController.jobCreate$original$8IQiGLKo(JobController.java:142) ~[classes!/:2.7.12]
at com.proinnova.system.job.rest.JobController.jobCreate$original$8IQiGLKo$accessor$q1MTONRX(JobController.java) ~[classes!/:2.7.12]
at com.proinnova.system.job.rest.JobController$auxiliary$xTO4F8Bi.call(Unknown Source) ~[classes!/:2.7.12]
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) [skywalking-agent.jar:8.13.0]
at com.proinnova.system.job.rest.JobController.jobCreate(JobController.java) ~[classes!/:2.7.12]
at com.proinnova.system.job.rest.JobController$$FastClassBySpringCGLIB$$22fc8d52.invoke() ~[classes!/:2.7.12]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:102) ~[spring-aop-5.3.27.jar!/:5.3.27]
at com.proinnova.system.dmjdemo.ControllerAspect.processLog(ControllerAspect.java:46) ~[classes!/:2.7.12]
at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0-272]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0-272]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.27.jar!/:5.3.27]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.27.jar!/:5.3.27]
at com.proinnova.system.job.rest.JobController$$EnhancerBySpringCGLIB$$e0d23851.jobCreate() ~[classes!/:2.7.12]
at sun.reflect.GeneratedMethodAccessor1219.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0-272]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0-272]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.27.jar!/:5.3.27]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:696) [tomcat-embed-core-9.0.71.jar!/:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.27.jar!/:5.3.27]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at com.seepln.common.rl.filter.RequestMonitorFilter.doFilterInternal(RequestMonitorFilter.java:65) [request-log-1.1.6.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.27.jar!/:5.3.27]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.27.jar!/:5.3.27]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.7.12.jar!/:2.7.12]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.27.jar!/:5.3.27]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.StandardHostValve.invoke$original$mcNA40pi(StandardHostValve.java:135) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.StandardHostValve.invoke$original$mcNA40pi$accessor$hv8NWlBb(StandardHostValve.java) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.StandardHostValve$auxiliary$ZMJ7BwmO.call(Unknown Source) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) [skywalking-agent.jar:8.13.0]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.71.jar!/:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.71.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0-272]
Caused by: java.lang.RuntimeException: 30,000 milliseconds timeout on connection http-outgoing-5603 [ACTIVE]
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:599) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12]
... 103 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5603 [ACTIVE]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:903) ~[elasticsearch-rest-client-7.17.3.jar!/:7.17.3]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:299) ~[elasticsearch-rest-client-7.17.3.jar!/:7.17.3]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:287) ~[elasticsearch-rest-client-7.17.3.jar!/:7.17.3]
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.3.jar!/:7.17.3]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.3.jar!/:7.17.3]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.3.jar!/:7.17.3]
at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:1279) ~[elasticsearch-rest-high-level-client-7.17.3.jar!/:7.17.3]
at org.springframework.data.elasticsearch.core.RestIndexTemplate.lambda$doExists$2(RestIndexTemplate.java:106) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12]
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:582) ~[spring-data-elasticsearch-4.4.12.jar!/:4.4.12]
... 102 common frames omitted
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5603 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:98) ~[httpasyncclient-4.1.5.jar!/:4.1.5]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar!/:4.1.5]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:506) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.16.jar!/:4.4.16]
... 1 common frames omitted
This is the elasticsearch server log: [2024-04-02T17:43:47,318][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve stats for node [zxt4RAOiRZy9Lol9IdIGfg]: [node_2][10.202.152.18:9300][cluster:monitor/nodes/stats[n]] request_id [772476 82] timed out after [15016ms] [2024-04-02T17:43:47,319][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve shard stats from node [zxt4RAOiRZy9Lol9IdIGfg]: [node_2][10.202.152.18:9300][indices:monitor/stats[n]] request_id [77247 683] timed out after [15016ms] [2024-04-02T17:43:47,319][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve shard stats from node [h4Igv51yStGQT552LJlITg]: [node_1][10.202.152.10:9300][indices:monitor/stats[n]] request_id [77247 684] timed out after [15016ms] [2024-04-02T17:43:47,320][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve shard stats from node [WSHeMx5fQbGTE_GtNc2fKg]: [node_3][10.202.152.15:9300][indices:monitor/stats[n]] request_id [77247 685] timed out after [15016ms] [2024-04-02T17:44:32,321][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve stats for node [WSHeMx5fQbGTE_GtNc2fKg]: [node_3][10.202.152.15:9300][cluster:monitor/nodes/stats[n]] request_id [772476 87] timed out after [15009ms] [2024-04-02T17:44:32,323][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve stats for node [zxt4RAOiRZy9Lol9IdIGfg]: [node_2][10.202.152.18:9300][cluster:monitor/nodes/stats[n]] request_id [772476 88] timed out after [15009ms] [2024-04-02T17:44:32,324][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve shard stats from node [zxt4RAOiRZy9Lol9IdIGfg]: [node_2][10.202.152.18:9300][indices:monitor/stats[n]] request_id [77247 689] timed out after [15009ms] [2024-04-02T17:44:32,324][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve shard stats from node [h4Igv51yStGQT552LJlITg]: [node_1][10.202.152.10:9300][indices:monitor/stats[n]] request_id [77247 690] timed out after [15009ms] [2024-04-02T17:44:32,324][WARN ][o.e.c.InternalClusterInfoService] [node_2] failed to retrieve shard stats from node [WSHeMx5fQbGTE_GtNc2fKg]: [node_3][10.202.152.15:9300][indices:monitor/stats[n]] request_id [77247 691] timed out after [15009ms] [2024-04-02T17:44:44,542][WARN ][o.e.t.TransportService ] [node_2] Received response for a request that has timed out, sent [7.2m/432269ms] ago, timed out [6.9m/417260ms] ago, action [indices:monitor/stats[n]], no de [{node_3}{WSHeMx5fQbGTE_GtNc2fKg}{j0aLmJ_1TnCAxNbdFUmuDw}{10.202.152.15}{10.202.152.15:9300}{cdfhilmrstw}{ml.machine_memory=16656986112, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1038876672, tra nsform.node=true}], id [77247566] [2024-04-02T17:44:44,543][WARN ][o.e.t.TransportService ] [node_2] Received response for a request that has timed out, sent [6.4m/387243ms] ago, timed out [6.2m/372235ms] ago, action [cluster:monitor/nodes/stats[n ]], node [{node_3}{WSHeMx5fQbGTE_GtNc2fKg}{j0aLmJ_1TnCAxNbdFUmuDw}{10.202.152.15}{10.202.152.15:9300}{cdfhilmrstw}{ml.machine_memory=16656986112, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=103887667 2, transform.node=true}], id [77247602] [2024-04-02T17:44:44,543][WARN ][o.e.t.TransportService ] [node_2] Received response for a request that has timed out, sent [2.7m/162310ms] ago, timed out [2.4m/147294ms] ago, action [cluster:monitor/nodes/stats[n ]], node [{node_3}{WSHeMx5fQbGTE_GtNc2fKg}{j0aLmJ_1TnCAxNbdFUmuDw}{10.202.152.15}{10.202.152.15:9300}{cdfhilmrstw}{ml.machine_memory=16656986112, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=103887667 2, transform.node=true}], id [77247669] [2024-04-02T17:44:44,585][WARN ][o.e.t.OutboundHandler ] [node_2] sending transport message [Request{indices:monitor/stats[n]}{77247654}{false}{true}{false}] of size [840] on [Netty4TcpChannel{localAddress=/10.20 2.152.18:51856, remoteAddress=10.202.152.10/10.202.152.10:9300, profile=default}] took [297189ms] which is above the warn threshold of [5000ms] with success [true]