search

pinpoint-apm / pinpoint-docker

Official Dockerized components of the Pinpoint
http://pinpoint-apm.github.io/pinpoint/
Apache License 2.0
461 stars 227 forks source link

Components loss hbase connection with gc problem #116

Open wonderfulwxw opened 3 years ago

wonderfulwxw commented 3 years ago

Hi, Pinpoint team,

We deployed pinpoint components with this repository on k8s, thanks for the efforts to supply these docker images, all of them works well after deployed.

But we found the webapp of pinpoint couldn't be accessed after two or three days... we checked the logs found that hbase has disconnected with webapp and zk

logs of webapp ``` java.lang.RuntimeException: java.io.IOException: Failed to get result within timeout, timeout=60000ms at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:95) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at com.navercorp.pinpoint.common.hbase.RowMapperResultsExtractor.extractData(RowMapperResultsExtractor.java:46) ~[pinpoint-commons-hbase-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.common.hbase.RowMapperResultsExtractor.extractData(RowMapperResultsExtractor.java:30) ~[pinpoint-commons-hbase-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2$1.doInTable(HbaseTemplate2.java:185) ~[pinpoint-commons-hbase-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.execute(HbaseTemplate2.java:751) ~[pinpoint-commons-hbase-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.find(HbaseTemplate2.java:180) ~[pinpoint-commons-hbase-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.find(HbaseTemplate2.java:195) ~[pinpoint-commons-hbase-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.web.dao.hbase.HbaseApplicationIndexDao.selectAllApplicationNames(HbaseApplicationIndexDao.java:75) ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.web.dao.hbase.HbaseApplicationIndexDao$$FastClassBySpringCGLIB$$22682794.invoke() ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) ~[spring-aop-5.3.6.jar!/:5.3.6] at com.navercorp.pinpoint.web.dao.hbase.HbaseApplicationIndexDao$$EnhancerBySpringCGLIB$$2ba14b81.selectAllApplicationNames() ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.web.service.CommonServiceImpl.selectAllApplicationNames(CommonServiceImpl.java:41) ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.web.controller.MainController.getApplicationGroup(MainController.java:49) ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at com.navercorp.pinpoint.web.controller.MainController$$FastClassBySpringCGLIB$$edd15fbb.invoke() ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) ~[spring-aop-5.3.6.jar!/:5.3.6] at com.navercorp.pinpoint.web.interceptor.PerformanceLoggingInterceptor.logging(PerformanceLoggingInterceptor.java:26) ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_272] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_272] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar!/:5.3.6] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) ~[spring-aop-5.3.6.jar!/:5.3.6] at com.navercorp.pinpoint.web.controller.MainController$$EnhancerBySpringCGLIB$$a14018c3.getApplicationGroup() ~[pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_272] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_272] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_272] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_272] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.6.jar!/:5.3.6] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060) [spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962) [spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.6.jar!/:5.3.6] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.3.6.jar!/:5.3.6] at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) [tomcat-embed-core-9.0.45.jar!/:4.0.1] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.6.jar!/:5.3.6] at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [tomcat-embed-core-9.0.45.jar!/:4.0.1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at com.navercorp.pinpoint.web.servlet.RewriteForV2Filter.doFilter(RewriteForV2Filter.java:92) [pinpoint-web-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at com.navercorp.pinpoint.web.servlet.NssFilter.doFilter(NssFilter.java:72) [naver-pinpoint-web-common-2.3.0-SNAPSHOT.jar!/:2.3.0-SNAPSHOT] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:106) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:149) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter.doFilterInternal(DefaultLogoutPageGeneratingFilter.java:58) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:237) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:223) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at com.navercorp.pinpoint.web.security.NssAuthenticationFilter.doFilterInternal(NssAuthenticationFilter.java:80) [classes!/:2.3.0-SNAPSHOT] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) [spring-security-web-5.4.6.jar!/:5.4.6] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.3.6.jar!/:5.3.6] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.6.jar!/:5.3.6] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.3.6.jar!/:5.3.6] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) [tomcat-embed-core-9.0.45.jar!/:?] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.45.jar!/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.45.jar!/:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272] Caused by: java.io.IOException: Failed to get result within timeout, timeout=60000ms at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:232) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:92) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] ... 142 more ```
logs of zk ``` 08-23 05:37:28.028 [tEvent-Worker-3] ERROR .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(com.navercorp.pinpoint.collector.service.async.AgentProperty,long,com.navercorp.pinpoint.common.server.util.AgentEventType) com.navercorp.pinpoint.common.hbase.HbaseSystemException: Failed 1 action: RetriesExhaustedException: 1 time, servers with issues: null; nested exception is org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: RetriesExhaustedException: 1 time, servers with issues: null at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.execute(HbaseTemplate2.java:713) ~[pinpoint-commons-hbase-2.2.0.jar!/:2.2.0] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.put(HbaseTemplate2.java:283) ~[pinpoint-commons-hbase-2.2.0.jar!/:2.2.0] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.put(HbaseTemplate2.java:278) ~[pinpoint-commons-hbase-2.2.0.jar!/:2.2.0] at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao.insert(HbaseAgentEventDao.java:81) ~[classes!/:2.2.0] at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$FastClassBySpringCGLIB$$28cd1468.invoke() ~[classes!/:2.2.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$EnhancerBySpringCGLIB$$d45f47ec.insert() ~[classes!/:2.2.0] at com.navercorp.pinpoint.collector.service.AgentEventService.insert(AgentEventService.java:42) ~[classes!/:2.2.0] at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(AgentEventAsyncTaskService.java:51) ~[classes!/:2.2.0] at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService$$FastClassBySpringCGLIB$$b2911b07.invoke() ~[classes!/:2.2.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) ~[spring-aop-5.2.7.RELEASE.jar!/:5.2.7.RELEASE] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: RetriesExhaustedException: 1 time, servers with issues: null at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:297) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2300(AsyncProcess.java:273) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1937) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:249) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:212) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1480) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1027) ~[hbase-shaded-client-1.4.13.jar!/:1.4.13] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2$6.doInTable(HbaseTemplate2.java:288) ~[pinpoint-commons-hbase-2.2.0.jar!/:2.2.0] at com.navercorp.pinpoint.common.hbase.HbaseTemplate2.execute(HbaseTemplate2.java:704) ~[pinpoint-commons-hbase-2.2.0.jar!/:2.2.0] ... 25 more 08-23 05:38:07.007 [d--pool19-t3292] INFO o.a.h.h.c.RpcRetryingCaller : Call exception, tries=10, retries=35, started=38192 ms ago, cancelled=false, msg=Call to pinpoint-hbase-0/ipipip:16201 failed on connection exception: java.net.ConnectException: Connection refused row 'AgentInfo,app-�����,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=pinpoint-hbase-0,16201,1629452703146, seqNum=0 ```

we didn't find the root cause why the hbase would loss its connection, then just restart the hbase, this problem would be solved. but after 2 days, hbase would lose the link again.

Some days later, we found there are some gc logs on hbase and the maxheapsize is not normal on hbase management web, it is too big ( 30G ), the pod max limitaion of mem only 8G. so we think here may has problem in configurtion.

gc logs - Alloction Failure ``` 2021-08-21T03:26:21.968+0900: [GC (Allocation Failure) 2021-08-21T03:26:21.968+0900: [ParNew: 606809K->54178K(629120K), 0.3459767 secs] 757495K->229108K(2027264K), 0.3461097 secs] [Times: user=1.55 sys=0.06, real=0.35 secs] 2021-08-21T03:35:21.960+0900: [GC (Allocation Failure) 2021-08-21T03:35:21.961+0900: [ParNew: 613410K->50776K(629120K), 0.1672801 secs] 788340K->254138K(2027264K), 0.1674248 secs] [Times: user=0.97 sys=0.05, real=0.17 secs] 2021-08-21T03:44:33.957+0900: [GC (Allocation Failure) 2021-08-21T03:44:33.957+0900: [ParNew: 610008K->36079K(629120K), 0.1636355 secs] 813370K->261776K(2027264K), 0.1637484 secs] [Times: user=0.91 sys=0.03, real=0.16 secs] 2021-08-21T03:53:02.059+0900: [GC (Allocation Failure) 2021-08-21T03:53:02.059+0900: [ParNew: 595311K->69888K(629120K), 0.0570013 secs] 821008K->295621K(2027264K), 0.0571376 secs] [Times: user=0.47 sys=0.00, real=0.06 secs] 2021-08-21T04:02:12.120+0900: [GC (Allocation Failure) 2021-08-21T04:02:12.120+0900: [ParNew: 629120K->68799K(629120K), 0.1004672 secs] 854853K->333086K(2027264K), 0.1006226 secs] [Times: user=0.38 sys=0.03, real=0.10 secs] 2021-08-21T04:11:17.948+0900: [GC (Allocation Failure) 2021-08-21T04:11:17.948+0900: [ParNew: 628031K->51988K(629120K), 0.1743066 secs] 892318K->332140K(2027264K), 0.1744972 secs] [Times: user=0.94 sys=0.04, real=0.17 secs] 2021-08-21T04:20:17.251+0900: [GC (Allocation Failure) 2021-08-21T04:20:17.251+0900: [ParNew: 611220K->35326K(629120K), 0.0776287 secs] 891372K->333913K(2027264K), 0.0777374 secs] [Times: user=0.68 sys=0.02, real=0.08 secs] 2021-08-21T04:29:31.926+0900: [GC (Allocation Failure) 2021-08-21T04:29:31.926+0900: [ParNew: 594558K->62205K(629120K), 0.0873964 secs] 893145K->360792K(2027264K), 0.0875510 secs] [Times: user=0.45 sys=0.00, real=0.09 secs] 2021-08-21T04:38:31.950+0900: [GC (Allocation Failure) 2021-08-21T04:38:31.950+0900: [ParNew: 621437K->69888K(629120K), 0.0727949 secs] 920024K->384187K(2027264K), 0.0729608 secs] [Times: user=0.45 sys=0.03, real=0.08 secs] 2021-08-21T04:47:11.914+0900: [GC (Allocation Failure) 2021-08-21T04:47:11.914+0900: [ParNew: 629120K->53902K(629120K), 0.2014880 secs] 943419K->405097K(2027264K), 0.2016107 secs] [Times: user=0.81 sys=0.04, real=0.20 secs] 2021-08-21T04:56:32.707+0900: [GC (Allocation Failure) 2021-08-21T04:56:32.707+0900: [ParNew: 613134K->41863K(629120K), 0.1145884 secs] 964329K->417241K(2027264K), 0.1147229 secs] [Times: user=0.63 sys=0.02, real=0.11 secs] 2021-08-21T05:05:13.446+0900: [GC (Allocation Failure) 2021-08-21T05:05:13.446+0900: [ParNew: 601095K->69888K(629120K), 0.1673223 secs] 976473K->449492K(2027264K), 0.1674816 secs] [Times: user=0.79 sys=0.02, real=0.17 secs] 2021-08-21T05:14:32.721+0900: [GC (Allocation Failure) 2021-08-21T05:14:32.721+0900: [ParNew: 629120K->69888K(629120K), 0.2056803 secs] 1008724K->464466K(2027264K), 0.2058106 secs] [Times: user=1.04 sys=0.08, real=0.21 secs] 2021-08-21T05:23:29.907+0900: [GC (Allocation Failure) 2021-08-21T05:23:29.907+0900: [ParNew: 629120K->63521K(629120K), 0.2118069 secs] 1023698K->484031K(2027264K), 0.2119219 secs] [Times: user=0.99 sys=0.01, real=0.21 secs] 2021-08-21T05:32:29.230+0900: [GC (Allocation Failure) 2021-08-21T05:32:29.230+0900: [ParNew: 622753K->40070K(629120K), 0.1832397 secs] 1043263K->482276K(2027264K), 0.1833736 secs] [Times: user=0.82 sys=0.01, real=0.19 secs] 2021-08-21T05:41:17.906+0900: [GC (Allocation Failure) 2021-08-21T05:41:17.906+0900: [ParNew: 599302K->69888K(629120K), 0.2116732 secs] 1041508K->519311K(2027264K), 0.2119217 secs] [Times: user=0.91 sys=0.02, real=0.21 secs] 2021-08-21T05:50:27.883+0900: [GC (Allocation Failure) 2021-08-21T05:50:27.883+0900: [ParNew: 629120K->51030K(629120K), 0.0597484 secs] 1078543K->543928K(2027264K), 0.0657957 secs] [Times: user=0.76 sys=0.03, real=0.07 secs] 2021-08-21T06:00:01.880+0900: [GC (Allocation Failure) 2021-08-21T06:00:01.880+0900: [ParNew: 610262K->41057K(629120K), 0.0275795 secs] 1103160K->556296K(2027264K), 0.0277075 secs] [Times: user=0.25 sys=0.03, real=0.03 secs] 2021-08-21T06:08:53.875+0900: [GC (Allocation Failure) 2021-08-21T06:08:53.875+0900: [ParNew: 600289K->69888K(629120K), 0.1488811 secs] 1115528K->592120K(2027264K), 0.1490273 secs] [Times: user=1.05 sys=0.01, real=0.15 secs] 2021-08-21T06:18:07.880+0900: [GC (Allocation Failure) 2021-08-21T06:18:07.880+0900: [ParNew: 629120K->50853K(629120K), 0.1332574 secs] 1151352K->595411K(2027264K), 0.1333634 secs] [Times: user=0.79 sys=0.02, real=0.14 secs] 2021-08-21T06:27:45.044+0900: [GC (Allocation Failure) 2021-08-21T06:27:45.044+0900: [ParNew: 610085K->32544K(629120K), 0.2759373 secs] 1154643K->601061K(2027264K), 0.2761310 secs] [Times: user=1.35 sys=0.01, real=0.27 secs] 2021-08-21T06:50:37.162+0900: [GC (Allocation Failure) 2021-08-21T06:50:37.162+0900: [ParNew: 591776K->69888K(629120K), 0.1642495 secs] 1160293K->654388K(2027264K), 0.1644342 secs] [Times: user=1.01 sys=0.05, real=0.17 secs] 2021-08-21T07:14:53.749+0900: [GC (Allocation Failure) 2021-08-21T07:14:53.749+0900: [ParNew: 629099K->57104K(629120K), 0.0790576 secs] 1213599K->693744K(2027264K), 0.0791955 secs] [Times: user=0.58 sys=0.06, real=0.08 secs] 2021-08-21T07:38:07.205+0900: [GC (Allocation Failure) 2021-08-21T07:38:07.205+0900: [ParNew: 616336K->69888K(629120K), 0.1146491 secs] 1252976K->762274K(2027264K), 0.1147720 secs] [Times: user=0.56 sys=0.04, real=0.11 secs] 2021-08-21T07:40:11.499+0900: [GC (Allocation Failure) 2021-08-21T07:40:11.499+0900: [ParNew: 629120K->69816K(629120K), 0.1261453 secs] 1321506K->1072014K(2027264K), 0.1262737 secs] [Times: user=0.69 sys=0.24, real=0.13 secs] 2021-08-21T07:40:11.626+0900: [GC (CMS Initial Mark) [1 CMS-initial-mark: 1002197K(1398144K)] 1072083K(2027264K), 0.0012808 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2021-08-21T07:40:11.628+0900: [CMS-concurrent-mark-start] 2021-08-21T07:40:11.726+0900: [CMS-concurrent-mark: 0.098/0.098 secs] [Times: user=0.23 sys=0.02, real=0.10 secs] 2021-08-21T07:40:11.726+0900: [CMS-concurrent-preclean-start] 2021-08-21T07:40:11.730+0900: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2021-08-21T07:40:11.731+0900: [GC (CMS Final Remark) [YG occupancy: 71142 K (629120 K)]2021-08-21T07:40:11.731+0900: [Rescan (parallel) , 0.0819049 secs]2021-08-21T07:40:11.813+0900: [weak refs processing, 0.0006648 secs]2021-08-21T07:40:11.814+0900: [class unloading, 0.0192887 secs]2021-08-21T07:40:11.833+0900: [scrub symbol table, 0.0079113 secs]2021-08-21T07:40:11.841+0900: [scrub string table, 0.0007006 secs][1 CMS-remark: 1002197K(1398144K)] 1073339K(2027264K), 0.1107974 secs] [Times: user=0.31 sys=0.01, real=0.11 secs] 2021-08-21T07:40:11.842+0900: [CMS-concurrent-sweep-start] 2021-08-21T07:40:12.438+0900: [CMS-concurrent-sweep: 0.596/0.596 secs] [Times: user=0.54 sys=0.07, real=0.60 secs] 2021-08-21T07:40:12.438+0900: [CMS-concurrent-reset-start] 2021-08-21T07:40:12.571+0900: [CMS-concurrent-reset: 0.133/0.133 secs] [Times: user=0.07 sys=0.08, real=0.13 secs] 2021-08-21T07:44:35.162+0900: [GC (Allocation Failure) 2021-08-21T07:44:35.162+0900: [ParNew: 627359K->69888K(629120K), 0.0742268 secs] 1077613K->803989K(2027264K), 0.0744741 secs] [Times: user=0.60 sys=0.17, real=0.08 secs] 2021-08-21T07:44:35.239+0900: [GC (CMS Initial Mark) [1 CMS-initial-mark: 734101K(1398144K)] 806204K(2027264K), 0.0016464 secs] [Times: user=0.01 sys=0.01, real=0.00 secs] 2021-08-21T07:44:35.240+0900: [CMS-concurrent-mark-start] 2021-08-21T07:44:35.331+0900: [CMS-concurrent-mark: 0.090/0.090 secs] [Times: user=0.18 sys=0.01, real=0.09 secs] 2021-08-21T07:44:35.331+0900: [CMS-concurrent-preclean-start] 2021-08-21T07:44:35.334+0900: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2021-08-21T07:44:35.334+0900: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 2021-08-21T07:44:40.420+0900: [CMS-concurrent-abortable-preclean: 1.182/5.086 secs] [Times: user=1.30 sys=0.00, real=5.08 secs] 2021-08-21T07:44:40.421+0900: [GC (CMS Final Remark) [YG occupancy: 89986 K (629120 K)]2021-08-21T07:44:40.421+0900: [Rescan (parallel) , 0.0916486 secs]2021-08-21T07:44:40.513+0900: [weak refs processing, 0.0000897 secs]2021-08-21T07:44:40.513+0900: [class unloading, 0.0233193 secs]2021-08-21T07:44:40.536+0900: [scrub symbol table, 0.0089437 secs]2021-08-21T07:44:40.545+0900: [scrub string table, 0.0009063 secs][1 CMS-remark: 734101K(1398144K)] 824088K(2027264K), 0.1251120 secs] [Times: user=0.45 sys=0.02, real=0.13 secs] 2021-08-21T07:44:40.546+0900: [CMS-concurrent-sweep-start] 2021-08-21T07:44:41.642+0900: [CMS-concurrent-sweep: 1.096/1.096 secs] [Times: user=1.05 sys=0.08, real=1.09 secs] 2021-08-21T07:44:41.643+0900: [CMS-concurrent-reset-start] 2021-08-21T07:44:41.782+0900: [CMS-concurrent-reset: 0.140/0.140 secs] [Times: user=0.07 sys=0.07, real=0.14 secs] ```

image

After checked the default configuration of hbase, there is no limitation of heap size.

default configs from dockerfile ``` # #/** # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information # * regarding copyright ownership. The ASF licenses this file # * to you under the Apache License, Version 2.0 (the # * "License"); you may not use this file except in compliance # * with the License. You may obtain a copy of the License at # * # * http://www.apache.org/licenses/LICENSE-2.0 # * # * Unless required by applicable law or agreed to in writing, software # * distributed under the License is distributed on an "AS IS" BASIS, # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # * See the License for the specific language governing permissions and # * limitations under the License. # */ # Set environment variables here. # This script sets variables multiple times over the course of starting an hbase process, # so try to keep things idempotent unless you want to take an even deeper look # into the startup scripts (bin/hbase, etc.) # The java implementation to use. Java 1.7+ required. # export JAVA_HOME=/usr/java/jdk1.6.0/ # Extra Java CLASSPATH elements. Optional. # export HBASE_CLASSPATH= # The maximum amount of heap to use. Default is left to JVM default. # export HBASE_HEAPSIZE=1G # Uncomment below if you intend to use off heap cache. For example, to allocate 8G of # offheap, set the value to "8G". # export HBASE_OFFHEAPSIZE=1G # Extra Java runtime options. # Below are what we set by default. May only work with SUN JVM. # For more on why as well as other possible settings, # see http://wiki.apache.org/hadoop/PerformanceTuning export HBASE_OPTS="-XX:+UseConcMarkSweepGC" # Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+ export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m" export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m" # Uncomment one of the below three options to enable java garbage collection logging for the server-side processes. # This enables basic gc logging to the .out file. # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps" # This enables basic gc logging to its own file. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:" # This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" # Uncomment one of the below three options to enable java garbage collection logging for the client processes. # This enables basic gc logging to the .out file. # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps" # This enables basic gc logging to its own file. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:" # This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" # See the package documentation for org.apache.hadoop.hbase.io.hfile for other configurations # needed setting up off-heap block caching. # Uncomment and adjust to enable JMX exporting # See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access. # More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html # NOTE: HBase provides an alternative JMX implementation to fix the random ports issue, please see JMX # section in HBase Reference Guide for instructions. # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" # export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101" # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102" # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103" # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104" # export HBASE_REST_OPTS="$HBASE_REST_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10105" # File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default. # export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers # Uncomment and adjust to keep all the Region Server pages mapped to be memory resident #HBASE_REGIONSERVER_MLOCK=true #HBASE_REGIONSERVER_UID="hbase" # File naming hosts on which backup HMaster will run. $HBASE_HOME/conf/backup-masters by default. # export HBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters # Extra ssh options. Empty by default. # export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR" # Where log files are stored. $HBASE_HOME/logs by default. # export HBASE_LOG_DIR=${HBASE_HOME}/logs # Enable remote JDWP debugging of major HBase processes. Meant for Core Developers # export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070" # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071" # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072" # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073" # A string representing this instance of hbase. $USER by default. # export HBASE_IDENT_STRING=$USER # The scheduling priority for daemon processes. See 'man nice'. # export HBASE_NICENESS=10 # The directory where pid files are stored. /tmp by default. # export HBASE_PID_DIR=/var/hadoop/pids # Seconds to sleep between slave commands. Unset by default. This # can be useful in large clusters, where, e.g., slave rsyncs can # otherwise arrive faster than the master can service them. # export HBASE_SLAVE_SLEEP=0.1 # Tell HBase whether it should manage it's own instance of Zookeeper or not. export HBASE_MANAGES_ZK=false # The default log rolling policy is RFA, where the log file is rolled as per the size defined for the # RFA appender. Please refer to the log4j.properties file to see more details on this appender. # In case one needs to do log rolling on a date change, one should set the environment property # HBASE_ROOT_LOGGER to ",DRFA". # For example: # HBASE_ROOT_LOGGER=INFO,DRFA # The reason for changing default to RFA is to avoid the boundary case of filling out disk space as # DRFA doesn't put any cap on the log size. Please refer to HBase-5655 for more context. ```

So we modified the JVM parameter of: -Xmx2g to HBASE_MASTER_OPTS and HBASE_REGIONSERVER_OPTS in hbase-env.sh

Now.. it works well. this problem won't be happened.

Question:

Does the maxHeapSize is the keypoint of this problem ? Why the default configs of hbase would be started with 30G maxHeapSize ? Could this be an error ? We are not sure if this solution is correct, could you please offer us some more informations or suggestions ?

emeroad commented 3 years ago

The pinpoint docker image was developed for testing.

It is recommended to deploy fully-distributed hbase for real environment.

wonderfulwxw commented 3 years ago

Alright, Thanks for your suggestion.