search

F43nd1r / Acrarium

A Backend for ACRA written in Kotlin using Spring Boot, Vaadin and MySQL
Apache License 2.0
203 stars 53 forks source link

Acrarium stuck and exhausted too much CPU? #200

Closed CodingFishWu closed 1 year ago

CodingFishWu commented 3 years ago

I'm using docker-compose to set up Acrarium. When I simply throw RuntimeException, everything runs well. But then I try another test. I manually ran a recursive function and caused a StackOverflow Exception, and it made Acraium stuck. I found the CPU usage of Acraium process is up to nealy 100%.

I guess maybe the log data was too large, and Acraium stuck when saving data to mysql.

This is the log may help

Hibernate: select version0_.`id` as id1_10_, version0_.`app_id` as app_id5_10_, version0_.`code` as code2_10_, version0_.`name` as name4_10_ from `version` version0_ where version0_.`code`=? and version0_.`app_id`=?
acrarium    | Hibernate: select stacktrace0_.`id` as id1_5_, stacktrace0_.`bug_id` as bug_id3_5_, stacktrace0_.`stacktrace` as stacktra2_5_, stacktrace0_.`version_id` as version_4_5_ from `stacktrace` stacktrace0_ cross join `version` version1_ cross join `bug` bug2_ where stacktrace0_.`version_id`=version1_.`id` and stacktrace0_.`bug_id`=bug2_.`id` and stacktrace0_.`stacktrace`=? and version1_.`code`=? and bug2_.`app_id`=?
acrarium    | 2021-01-29 09:19:48.390  WARN 1 --- [nio-8080-exec-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: S1000
acrarium    | 2021-01-29 09:19:48.390 ERROR 1 --- [nio-8080-exec-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : Packet for query is too large (17,551,725 > 4,194,304). You can change this value on the server by setting the 'max_allowed_packet' variable.
acrarium    | 2021-01-29 09:19:48.413 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet] with root cause
acrarium    |
acrarium    | com.mysql.cj.jdbc.exceptions.PacketTooBigException: Packet for query is too large (17,551,725 > 4,194,304). You can change this value on the server by setting the 'max_allowed_packet' variable.
acrarium    |   at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:107) ~[mysql-connector-java-8.0.22.jar:8.0.22]
acrarium    |   at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) ~[mysql-connector-java-8.0.22.jar:8.0.22]
acrarium    |   at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003) ~[mysql-connector-java-8.0.22.jar:8.0.22]
acrarium    |   at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) ~[HikariCP-3.4.5.jar:na]
acrarium    |   at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) ~[HikariCP-3.4.5.jar:na]
acrarium    |   at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.getResultSet(Loader.java:2304) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2057) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2019) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.doQuery(Loader.java:948) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.doList(Loader.java:2850) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.doList(Loader.java:2832) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.Loader.list(Loader.java:2659) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1641) ~[hibernate-core-5.4.25.Final.jar:5.4.25.Final]
acrarium    |   at com.querydsl.jpa.impl.AbstractJPAQuery.getSingleResult(AbstractJPAQuery.java:183) ~[querydsl-jpa-4.4.0.jar:na]
acrarium    |   at com.querydsl.jpa.impl.AbstractJPAQuery.fetchOne(AbstractJPAQuery.java:253) ~[querydsl-jpa-4.4.0.jar:na]
acrarium    |   at com.faendir.acra.service.DataService.findStacktrace(DataService.kt:248) ~[classes/:1.1.0]
acrarium    |   at com.faendir.acra.service.DataService.createNewReport(DataService.kt:312) ~[classes/:1.1.0]
acrarium    |   at com.faendir.acra.service.DataService$$FastClassBySpringCGLIB$$c0f33e7.invoke(<generated>) ~[classes/:1.1.0]
acrarium    |   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) ~[spring-security-core-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at com.faendir.acra.service.DataService$$EnhancerBySpringCGLIB$$4524a548.createNewReport(<generated>) ~[classes/:1.1.0]
acrarium    |   at com.faendir.acra.rest.RestReportInterface.report(RestReportInterface.kt:46) ~[classes/:1.1.0]
acrarium    |   at com.faendir.acra.rest.RestReportInterface$$FastClassBySpringCGLIB$$710fedc2.invoke(<generated>) ~[classes/:1.1.0]
acrarium    |   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) ~[spring-security-core-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) ~[spring-aop-5.3.2.jar:5.3.2]
acrarium    |   at com.faendir.acra.rest.RestReportInterface$$EnhancerBySpringCGLIB$$38d99d05.report(<generated>) ~[classes/:1.1.0]
acrarium    |   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
acrarium    |   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
acrarium    |   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
acrarium    |   at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
acrarium    |   at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1061) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:961) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) ~[javax.servlet-api-4.0.1.jar:4.0.1]
acrarium    |   at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.2.jar:5.3.2]
acrarium    |   at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar:4.0.1]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263) ~[ziplet-2.4.3.jar:na]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:178) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) ~[spring-security-web-5.4.2.jar:5.4.2]
acrarium    |   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) ~[spring-boot-actuator-2.4.1.jar:2.4.1]
acrarium    |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.2.jar:5.3.2]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
acrarium    |   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
acrarium    |   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.41.jar:9.0.41]
acrarium    |   at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
acrarium    |
acrarium    | Hibernate: select user0_.`username` as username1_7_, user0_.`mail` as mail2_7_, user0_.`password` as password3_7_, permission2_.`user_username` as user_use1_8_1__, permission2_.`app_id` as app_id2_8_1__, permission2_.`level` as level3_8_1__, roles1_.`user_username` as user_use1_9_0__, roles1_.`roles` as roles2_9_0__ from `user` user0_ left outer join `user_roles` roles1_ on user0_.`username`=roles1_.`user_username` left outer join `user_permissions` permission2_ on user0_.`username`=permission2_.`user_username` where lower(user0_.`username`)=?
acrarium    | Hibernate: select app0_.`id` as id1_0_, app0_.`min_score` as min_scor2_0_, app0_.`name` as name3_0_, app0_.`reporter_username` as reporter4_0_ from `app` app0_ where app0_.`reporter_username`=?
acrarium    | Hibernate: select user0_.`username` as username1_7_0_, user0_.`mail` as mail2_7_0_, user0_.`password` as password3_7_0_, permission1_.`user_username` as user_use1_8_1_, permission1_.`app_id` as app_id2_8_1_, permission1_.`level` as level3_8_1_, app2_.`id` as id1_0_2_, app2_.`min_score` as min_scor2_0_2_, app2_.`name` as name3_0_2_, app2_.`reporter_username` as reporter4_0_2_, roles3_.`user_username` as user_use1_9_3_, roles3_.`roles` as roles2_9_3_ from `user` user0_ left outer join `user_permissions` permission1_ on user0_.`username`=permission1_.`user_username` left outer join `app` app2_ on permission1_.`app_id`=app2_.`id` left outer join `user_roles` roles3_ on user0_.`username`=roles3_.`user_username` where user0_.`username`=?
acrarium    | Hibernate: select version0_.`id` as id1_10_, version0_.`app_id` as app_id5_10_, version0_.`code` as code2_10_, version0_.`name` as name4_10_ from `version` version0_ where version0_.`code`=? and version0_.`app_id`=?
acrarium    | Hibernate: select stacktrace0_.`id` as id1_5_, stacktrace0_.`bug_id` as bug_id3_5_, stacktrace0_.`stacktrace` as stacktra2_5_, stacktrace0_.`version_id` as version_4_5_ from `stacktrace` stacktrace0_ cross join `version` version1_ cross join `bug` bug2_ where stacktrace0_.`version_id`=version1_.`id` and stacktrace0_.`bug_id`=bug2_.`id` and stacktrace0_.`stacktrace`=? and version1_.`code`=? and bug2_.`app_id`=?
acrarium    | Hibernate: select stacktrace0_.`id` as id1_5_, stacktrace0_.`bug_id` as bug_id3_5_, stacktrace0_.`stacktrace` as stacktra2_5_, stacktrace0_.`version_id` as version_4_5_ from `stacktrace` stacktrace0_ cross join `version` version1_ cross join `bug` bug2_ where stacktrace0_.`version_id`=version1_.`id` and stacktrace0_.`bug_id`=bug2_.`id` and stacktrace0_.`stacktrace`=? and version1_.`code`=? and bug2_.`app_id`=?
acrarium    | Hibernate: select bug1_.`id` as id1_2_0_, version2_.`id` as id1_10_1_, bug1_.`app_id` as app_id3_2_0_, bug1_.`solved_version` as solved_v4_2_0_, bug1_.`title` as title2_2_0_, version2_.`app_id` as app_id5_10_1_, version2_.`code` as code2_10_1_, version2_.`name` as name4_10_1_ from `stacktrace` stacktrace0_ inner join `bug` bug1_ on stacktrace0_.`bug_id`=bug1_.`id` left outer join `version` version2_ on bug1_.`solved_version`=version2_.`id` where bug1_.`app_id`=? and (stacktrace0_.`stacktrace` like ? escape '!') limit ?
acrarium    | Hibernate: insert into `bug` (`app_id`, `solved_version`, `title`) values (?, ?, ?)
acrarium    | Hibernate: insert into `stacktrace` (`bug_id`, `stacktrace`, `version_id`) values (?, ?, ?)
acrarium    | Hibernate: select report0_.`id` as id1_4_3_, report0_.`android_version` as android_2_4_3_, report0_.`brand` as brand3_4_3_, report0_.`content` as content4_4_3_, report0_.`date` as date5_4_3_, report0_.`installation_id` as installa6_4_3_, report0_.`phone_model` as phone_mo7_4_3_, report0_.`stacktrace_id` as stacktr10_4_3_, report0_.`user_comment` as user_com8_4_3_, report0_.`user_email` as user_ema9_4_3_, stacktrace1_.`id` as id1_5_0_, stacktrace1_.`bug_id` as bug_id3_5_0_, stacktrace1_.`stacktrace` as stacktra2_5_0_, stacktrace1_.`version_id` as version_4_5_0_, bug2_.`id` as id1_2_1_, bug2_.`app_id` as app_id3_2_1_, bug2_.`solved_version` as solved_v4_2_1_, bug2_.`title` as title2_2_1_, version3_.`id` as id1_10_2_, version3_.`app_id` as app_id5_10_2_, version3_.`code` as code2_10_2_, version3_.`name` as name4_10_2_ from `report` report0_ inner join `stacktrace` stacktrace1_ on report0_.`stacktrace_id`=stacktrace1_.`id` left outer join `bug` bug2_ on stacktrace1_.`bug_id`=bug2_.`id` left outer join `version` version3_ on stacktrace1_.`version_id`=version3_.`id` where report0_.`id`=?
acrarium    | Hibernate: insert into `report` (`android_version`, `brand`, `content`, `date`, `installation_id`, `phone_model`, `stacktrace_id`, `user_comment`, `user_email`, `id`) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
acrarium    | Hibernate: select bug1_.`id` as id1_2_0_, app2_.`id` as id1_0_1_, bug1_.`app_id` as app_id3_2_0_, bug1_.`solved_version` as solved_v4_2_0_, bug1_.`title` as title2_2_0_, app2_.`min_score` as min_scor2_0_1_, app2_.`name` as name3_0_1_, app2_.`reporter_username` as reporter4_0_1_ from `stacktrace` stacktrace0_ inner join `bug` bug1_ on stacktrace0_.`bug_id`=bug1_.`id` inner join `app` app2_ on bug1_.`app_id`=app2_.`id` where stacktrace0_.`id`=?
acrarium    | Hibernate: select stacktrace0_.`id` as id1_5_0_, bug1_.`id` as id1_2_1_, stacktrace0_.`bug_id` as bug_id3_5_0_, stacktrace0_.`stacktrace` as stacktra2_5_0_, stacktrace0_.`version_id` as version_4_5_0_, bug1_.`app_id` as app_id3_2_1_, bug1_.`solved_version` as solved_v4_2_1_, bug1_.`title` as title2_2_1_ from `stacktrace` stacktrace0_ inner join `bug` bug1_ on stacktrace0_.`bug_id`=bug1_.`id` where stacktrace0_.`id`<>? and bug1_.`app_id`=(select app5_.`id` from `stacktrace` stacktrace3_ inner join `bug` bug4_ on stacktrace3_.`bug_id`=bug4_.`id` inner join `app` app5_ on bug4_.`app_id`=app5_.`id` where stacktrace3_.`id`=?) and (stacktrace0_.`id` not in  (select stacktrace6_.`right_id` from `stacktrace_match` stacktrace6_ cross join `stacktrace` stacktrace7_ where stacktrace6_.`right_id`=stacktrace7_.`id` and stacktrace6_.`left_id`=?)) and (stacktrace0_.`id` not in  (select stacktrace8_.`left_id` from `stacktrace_match` stacktrace8_ cross join `stacktrace` stacktrace9_ where stacktrace8_.`left_id`=stacktrace9_.`id` and stacktrace8_.`right_id`=?))
acrarium    | Hibernate: select stacktrace0_.`left_id` as left_id3_6_0_, stacktrace0_.`right_id` as right_id2_6_0_, stacktrace0_.`score` as score1_6_0_ from `stacktrace_match` stacktrace0_ where stacktrace0_.`left_id`=? and stacktrace0_.`right_id`=?
acrarium    | Hibernate: select stacktrace0_.`left_id` as left_id3_6_0_, stacktrace0_.`right_id` as right_id2_6_0_, stacktrace0_.`score` as score1_6_0_ from `stacktrace_match` stacktrace0_ where stacktrace0_.`left_id`=? and stacktrace0_.`right_id`=?
CodingFishWu commented 3 years ago

I have tried add max_allowed_packet to mysql's starting command. But it shows another error.

Can anyone tell me how to solve this?

F43nd1r commented 3 years ago

it shows another error

Then post that error.

CodingFishWu commented 3 years ago

I mistook it. It didn't show any error. But it simply stuck with high cpu usage just after I post logs from the client. These is the logs I guess:

acrarium    | Hibernate: insert into `report` (`android_version`, `brand`, `content`, `date`, `installation_id`, `phone_model`, `stacktrace_id`, `user_comment`, `user_email`, `id`) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
acrarium    | Hibernate: select bug1_.`id` as id1_2_0_, app2_.`id` as id1_0_1_, bug1_.`app_id` as app_id3_2_0_, bug1_.`solved_version` as solved_v4_2_0_, bug1_.`title` as title2_2_0_, app2_.`min_score` as min_scor2_0_1_, app2_.`name` as name3_0_1_, app2_.`reporter_username` as reporter4_0_1_ from `stacktrace` stacktrace0_ inner join `bug` bug1_ on stacktrace0_.`bug_id`=bug1_.`id` inner join `app` app2_ on bug1_.`app_id`=app2_.`id` where stacktrace0_.`id`=?
acrarium    | Hibernate: select stacktrace0_.`id` as id1_5_0_, bug1_.`id` as id1_2_1_, stacktrace0_.`bug_id` as bug_id3_5_0_, stacktrace0_.`stacktrace` as stacktra2_5_0_, stacktrace0_.`version_id` as version_4_5_0_, bug1_.`app_id` as app_id3_2_1_, bug1_.`solved_version` as solved_v4_2_1_, bug1_.`title` as title2_2_1_ from `stacktrace` stacktrace0_ inner join `bug` bug1_ on stacktrace0_.`bug_id`=bug1_.`id` where stacktrace0_.`id`<>? and bug1_.`app_id`=(select app5_.`id` from `stacktrace` stacktrace3_ inner join `bug` bug4_ on stacktrace3_.`bug_id`=bug4_.`id` inner join `app` app5_ on bug4_.`app_id`=app5_.`id` where stacktrace3_.`id`=?) and (stacktrace0_.`id` not in  (select stacktrace6_.`right_id` from `stacktrace_match` stacktrace6_ cross join `stacktrace` stacktrace7_ where stacktrace6_.`right_id`=stacktrace7_.`id` and stacktrace6_.`left_id`=?)) and (stacktrace0_.`id` not in  (select stacktrace8_.`left_id` from `stacktrace_match` stacktrace8_ cross join `stacktrace` stacktrace9_ where stacktrace8_.`left_id`=stacktrace9_.`id` and stacktrace8_.`right_id`=?))
acrarium    | Hibernate: select stacktrace0_.`left_id` as left_id3_6_0_, stacktrace0_.`right_id` as right_id2_6_0_, stacktrace0_.`score` as score1_6_0_ from `stacktrace_match` stacktrace0_ where stacktrace0_.`left_id`=? and stacktrace0_.`right_id`=?
acrarium    | Hibernate: select stacktrace0_.`left_id` as left_id3_6_0_, stacktrace0_.`right_id` as right_id2_6_0_, stacktrace0_.`score` as score1_6_0_ from `stacktrace_match` stacktrace0_ where stacktrace0_.`left_id`=? and stacktrace0_.`right_id`=?
acrarium    | Hibernate: select user0_.`username` as username1_7_, user0_.`mail` as mail2_7_, user0_.`password` as password3_7_, permission2_.`user_username` as user_use1_8_1__, permission2_.`app_id` as app_id2_8_1__, permission2_.`level` as level3_8_1__, roles1_.`user_username` as user_use1_9_0__, roles1_.`roles` as roles2_9_0__ from `user` user0_ left outer join `user_roles` roles1_ on user0_.`username`=roles1_.`user_username` left outer join `user_permissions` permission2_ on user0_.`username`=permission2_.`user_username` where lower(user0_.`username`)=?
CodingFishWu commented 3 years ago

This is the code I'm using on client. It's quite simple.

        button.setOnClickListener(new Button.OnClickListener() {
            public void onClick(View v) {
                infinite();

            }
        });
    }

    private void infinite() {
        infinite();
    }

there's no errors shown in the console. But the application is stuck with high cpu usages forever.

JaimeToca commented 3 years ago

I'm facing a similar problem. The thing is that I checked the nginx logs and saw no requests at that time but there were 4 cpu's working 100%. It seems that it gets stuck in a infinite loop. @F43nd1r

This morning I create an empty D.B and today at 18:00 got the issue. Tried with both Docker and standalone. It happened after ~ 2k reports

Cheers

CodingFishWu commented 3 years ago

@F43nd1r Hi, can you reproduce the problem? I'm waiting for the solution, and I can give you more test cases if possible.

F43nd1r commented 3 years ago

I was unable to reproduce the problem. I tried a simple stackoverflow like the example and had no issues receiving or viewing it in Acrarium.

F43nd1r commented 3 years ago

I have a hunch what might be the issue here. @JaimeToca @CodingFishWu does this only happen after a certain number of reports have come in? If so, please report back with the size of your stacktrace_match table in your database.

a-kazakov commented 3 years ago

I have this happening very often starting from about a month ago. Is there anything I can do to help with investigating the issue? I'm currently running Acrarium as a Docker container.

F43nd1r commented 3 years ago

does this only happen after a certain number of reports have come in? If so, please report back with the size of your stacktrace_match table in your database.

@a-kazakov you can answer this.

a-kazakov commented 3 years ago

Yes, this does happen after some number of reports.

mysql> select count(*) from `stacktrace_match`;
+----------+
| count(*) |
+----------+
|  1176688 |
+----------+
1 row in set (0.12 sec)
kubasienki commented 3 years ago

It happens also for me - I wasn't even able to connect via ssh to my server. Running acrarium as a docker container. But I have 80 reports only. Interactions with gui seem to increase possibility of the bug occurence.

a-kazakov commented 3 years ago

As a workaround, I disconnected my Acrarium container from the internet, restarted it, then logged into admin panel and removed old reports. Seems like that helped. Without disconnecting it from the internet Acrarium got stuck before old reports removal completed.

mkocus commented 2 years ago

Got the same issue - cannot use Acrarium at all since about 2 weeks, because it constantly uses 100% CPU. I've tried to increase the amount of vCPU and RAM, increase the amount of max DB connections (both on mysql and HikariPool side), increase max packet for mysql, but with no avail.

Why on any moment of time only one mysql query can happen?

mysql> show processlist;
+------+----------+------------------+----------+---------+------+----------+------------------------------------------------------------------------------------------------------+
| Id   | User     | Host             | db       | Command | Time | State    | Info                                                                                                 |
+------+----------+------------------+----------+---------+------+----------+------------------------------------------------------------------------------------------------------+
| 1252 | acrarium | 172.21.0.3:43168 | acrarium | Sleep   |    1 |          | NULL                                                                                                 |
| 1254 | acrarium | 172.21.0.3:43172 | acrarium | Sleep   |   52 |          | NULL                                                                                                 |
| 1259 | acrarium | 172.21.0.3:43182 | acrarium | Sleep   |   37 |          | NULL                                                                                                 |
| 1261 | acrarium | 172.21.0.3:43186 | acrarium | Sleep   |   38 |          | NULL                                                                                                 |
| 1262 | acrarium | 172.21.0.3:43188 | acrarium | Sleep   |    5 |          | NULL                                                                                                 |
| 1263 | acrarium | 172.21.0.3:43190 | acrarium | Sleep   |   22 |          | NULL                                                                                                 |
| 1264 | acrarium | 172.21.0.3:43192 | acrarium | Sleep   |    7 |          | NULL                                                                                                 |
| 1265 | acrarium | 172.21.0.3:43194 | acrarium | Sleep   |    2 |          | NULL                                                                                                 |
| 1266 | acrarium | 172.21.0.3:43196 | acrarium | Sleep   |   54 |          | NULL                                                                                                 |
| 1267 | acrarium | 172.21.0.3:43198 | acrarium | Sleep   |   45 |          | NULL                                                                                                 |
| 1268 | acrarium | 172.21.0.3:43200 | acrarium | Sleep   |   56 |          | NULL                                                                                                 |
| 1269 | acrarium | 172.21.0.3:43202 | acrarium | Sleep   |   26 |          | NULL                                                                                                 |
| 1270 | acrarium | 172.21.0.3:43204 | acrarium | Sleep   |    4 |          | NULL                                                                                                 |
| 1271 | acrarium | 172.21.0.3:43206 | acrarium | Sleep   |   16 |          | NULL                                                                                                 |
| 1272 | acrarium | 172.21.0.3:43208 | acrarium | Sleep   |   32 |          | NULL                                                                                                 |
| 1273 | acrarium | 172.21.0.3:43210 | acrarium | Sleep   |   46 |          | NULL                                                                                                 |
| 1274 | acrarium | 172.21.0.3:43212 | acrarium | Sleep   |   21 |          | NULL                                                                                                 |
| 1275 | acrarium | 172.21.0.3:43216 | acrarium | Sleep   |   55 |          | NULL                                                                                                 |
| 1276 | acrarium | 172.21.0.3:43218 | acrarium | Sleep   |   14 |          | NULL                                                                                                 |
| 1277 | acrarium | 172.21.0.3:43220 | acrarium | Sleep   |   27 |          | NULL                                                                                                 |
| 1278 | acrarium | 172.21.0.3:43222 | acrarium | Sleep   |    1 |          | NULL                                                                                                 |
| 1279 | acrarium | 172.21.0.3:43224 | acrarium | Sleep   |   11 |          | NULL                                                                                                 |
| 1280 | acrarium | 172.21.0.3:43226 | acrarium | Sleep   |   39 |          | NULL                                                                                                 |
| 1281 | acrarium | 172.21.0.3:43228 | acrarium | Sleep   |   43 |          | NULL                                                                                                 |
| 1282 | acrarium | 172.21.0.3:43230 | acrarium | Sleep   |    8 |          | NULL                                                                                                 |
| 1283 | acrarium | 172.21.0.3:43232 | acrarium | Sleep   |    1 |          | NULL                                                                                                 |
| 1284 | acrarium | 172.21.0.3:43234 | acrarium | Sleep   |    5 |          | NULL                                                                                                 |
| 1285 | acrarium | 172.21.0.3:43236 | acrarium | Sleep   |   23 |          | NULL                                                                                                 |
| 1286 | acrarium | 172.21.0.3:43238 | acrarium | Sleep   |   29 |          | NULL                                                                                                 |
| 1287 | acrarium | 172.21.0.3:43240 | acrarium | Sleep   |   16 |          | NULL                                                                                                 |
| 1288 | acrarium | 172.21.0.3:43242 | acrarium | Sleep   |   20 |          | NULL                                                                                                 |
| 1289 | acrarium | 172.21.0.3:43244 | acrarium | Sleep   |    2 |          | NULL                                                                                                 |
| 1290 | acrarium | 172.21.0.3:43246 | acrarium | Sleep   |    6 |          | NULL                                                                                                 |
| 1291 | acrarium | 172.21.0.3:43248 | acrarium | Sleep   |   13 |          | NULL                                                                                                 |
| 1292 | acrarium | 172.21.0.3:43250 | acrarium | Sleep   |   54 |          | NULL                                                                                                 |
| 1293 | acrarium | 172.21.0.3:43252 | acrarium | Sleep   |    3 |          | NULL                                                                                                 |
| 1294 | acrarium | 172.21.0.3:43254 | acrarium | Sleep   |    3 |          | NULL                                                                                                 |
| 1295 | acrarium | 172.21.0.3:43256 | acrarium | Sleep   |   51 |          | NULL                                                                                                 |
| 1296 | acrarium | 172.21.0.3:43258 | acrarium | Sleep   |   78 |          | NULL                                                                                                 |
| 1297 | acrarium | 172.21.0.3:43260 | acrarium | Sleep   |    6 |          | NULL                                                                                                 |
| 1298 | acrarium | 172.21.0.3:43262 | acrarium | Sleep   |    0 |          | NULL                                                                                                 |
| 1299 | acrarium | 172.21.0.3:43264 | acrarium | Sleep   |   11 |          | NULL                                                                                                 |
| 1300 | acrarium | 172.21.0.3:43266 | acrarium | Sleep   |   63 |          | NULL                                                                                                 |
| 1301 | root     | localhost        | NULL     | Query   |   -1 | starting | show processlist                                                                                     |
| 1302 | acrarium | 172.21.0.3:43268 | acrarium | Sleep   |   20 |          | NULL                                                                                                 |
| 1303 | acrarium | 172.21.0.3:43270 | acrarium | Sleep   |   25 |          | NULL                                                                                                 |
| 1304 | acrarium | 172.21.0.3:43272 | acrarium | Sleep   |    2 |          | NULL                                                                                                 |
| 1305 | acrarium | 172.21.0.3:43274 | acrarium | Sleep   |    3 |          | NULL                                                                                                 |
| 1306 | acrarium | 172.21.0.3:43276 | acrarium | Query   |   30 | update   | insert into `stacktrace` (`bug_id`, `class`, `stacktrace`, `version_id`) values (4300, 'com.google.g |
| 1307 | acrarium | 172.21.0.3:43278 | acrarium | Sleep   |    7 |          | NULL                                                                                                 |
| 1308 | acrarium | 172.21.0.3:43280 | acrarium | Sleep   |    4 |          | NULL                                                                                                 |
+------+----------+------------------+----------+---------+------+----------+------------------------------------------------------------------------------------------------------+
51 rows in set (0.00 sec)

Acrarium container logs:

2021-12-17 09:51:00.375 ERROR 1 --- [o-8080-exec-239] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-1 - Connection is not available, request timed out after 30000ms.
2021-12-17 09:51:00.377 ERROR 1 --- [o-8080-exec-239] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:695) ~[HikariCP-3.4.5.jar:na]
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) ~[HikariCP-3.4.5.jar:na]
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) ~[HikariCP-3.4.5.jar:na]
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128) ~[HikariCP-3.4.5.jar:na]
        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:274) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:282) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83) ~[hibernate-core-5.4.30.Final.jar:5.4.30.Final]
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:164) ~[spring-orm-5.3.6.jar:5.3.6]
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421) ~[spring-orm-5.3.6.jar:5.3.6]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) ~[spring-tx-5.3.6.jar:5.3.6]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-5.3.6.jar:5.3.6]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) ~[spring-tx-5.3.6.jar:5.3.6]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) ~[spring-tx-5.3.6.jar:5.3.6]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) ~[spring-security-core-5.4.6.jar:5.4.6]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) ~[spring-aop-5.3.6.jar:5.3.6]
        at com.faendir.acra.service.DataService$$EnhancerBySpringCGLIB$$41261d92.createNewReport(<generated>) ~[classes/:1.4.0-beta4]
        at com.faendir.acra.rest.RestReportInterface.report(RestReportInterface.kt:45) ~[classes/:1.4.0-beta4]
        at com.faendir.acra.rest.RestReportInterface$$FastClassBySpringCGLIB$$710fedc2.invoke(<generated>) ~[classes/:1.4.0-beta4]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) ~[spring-security-core-5.4.6.jar:5.4.6]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750) ~[spring-aop-5.3.6.jar:5.3.6]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692) ~[spring-aop-5.3.6.jar:5.3.6]
        at com.faendir.acra.rest.RestReportInterface$$EnhancerBySpringCGLIB$$27c289f8.report(<generated>) ~[classes/:1.4.0-beta4]
        at jdk.internal.reflect.GeneratedMethodAccessor105.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) ~[tomcat-embed-core-9.0.45.jar:4.0.FR]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.6.jar:5.3.6]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.45.jar:4.0.FR]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263) ~[ziplet-2.4.3.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:178) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) ~[spring-security-web-5.4.6.jar:5.4.6]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) ~[spring-web-5.3.6.jar:5.3.6]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) ~[spring-boot-actuator-2.4.5.jar:2.4.5]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.6.jar:5.3.6]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

2021-12-17 09:51:02.535  WARN 1 --- [o-8080-exec-290] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is org.apache.catalina.connector.ClientAbortException: java.io.IOException: Unable to unwrap data, invalid status [CLOSED]]

I started by using standard docker-compose config from setup guide, then I changed the parameters mentioned earlier.

Do you have any idea?

barisahmet commented 2 years ago

I have the same issue. I have an app with more than 10M users, and getting a lot of ACRA reports as you may guess. After 80 - 100 reports, it stucks at 100% cpu usage.

Tried to put my acrarium instance behind HaProxy with a maximum 1 request per second to see if it get stucks because of request spike. However it still stucks at ~100 reports.

Logs are similar to what others posted. HikariPool tells "no connection available".

F43nd1r commented 2 years ago

Acrarium was not really built for high report counts, as it was just for my small personal projects.

There is a linear call on previous records when receiving a new record which takes too long on large databases, i could reproduce this upwards of 10000 reports. You can a) remove old reports from your database b) update to 1.7.0 and set the matching score to 100 (disables any bug auto merging except exact match, removes linear call) c) get me someone who understands big data so we can figure out how to do this correctly.

mikehardy commented 2 years ago

This one just bit ankidroid.org on a pretty low count of exceptions - just around 900 reports - I think the problem gets worse as you use lower-powered servers because on creation of a new report the fuzzy match takes long enough that the client disconnects but then retries. So each client then becomes a little DoS factory pinning the server almost immediately.

Setting match to 100% mitigated this completely for us.

F43nd1r commented 2 years ago

Good to know the mitigation works :+1:

mikehardy commented 2 years ago

I believe the standard method here would be to make bug fuzzy match non-critical path since it's an essentially infinite sink of work. Stated differently, report ingress right now blocks on the match/coalesce I believe? So instead reports would get a status column to track match status (like "yes matched", "no not matched" and/or "in progress, worker XYZ") and there would be a queue you'd add to after letting the ACRA client go with an HTTP 200 (and maybe you could manually trigger a re-match for an existing issue or a multi-select or all issues).

Then as you fork off that thread you set min priority, with documentation (or maybe a check and warning log?) that the JVM is running as root on linux so that it's possible for thread priorities to work:

https://stackoverflow.com/a/1662571/9910298

Still valid for JDK11 on linux: https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/hotspot/os/linux/os_linux.cpp#L4136-L4171

Now you can get fuzzy matching, saturate the CPUs doing it if you want, but the server will still be interactive and client's won't death spiral the server on timeout/resubmits

mikehardy commented 2 years ago

I have so much open source stuff on my plate already I won't have time to pursue this, but that's my big-ish data data on it: separate tasks cleanly between "interactive" and "processing" and never mix the two basically, then make sure interactive always gets the time slice when things are competitve.

F43nd1r commented 2 years ago

Switching the matching to a long running asynchronous task brings some problems that I don't really want to deal with, but regardless: I think the correct way forward is to design a good fingerprinting algorithm for stacktraces such that a database index can be used instead of the linear fuzzy search

my development time is very limited currently though so don't expect anything to happen.

mikehardy commented 2 years ago

I'm the same (time-restricted) but knowing what sort of PR you would allow (pre-compute hash(es) + index those + use that for merging - is that right?) may help direct someone/anyone/my-future-self ;-).

There is some prior art https://rp.os3.nl/2020-2021/p60/report.pdf

Seems like having multiple hashes may be interesting. Could hash first N characters of exception message, could have configuration for code package name of the app, and hash complete line of first stack frame with configured application name, could hash the whole stack trace (within reason I guess, whole NN chars of whole stack frame). Then critical path performance would be O(1) for hash, and O(logN or whatever) for sql lookup and that's it? The big-data answer to the O(logN) thing as N gets really huge is map/reduce and database shards but it's Acrarium here not Firebase crashlytics ;-)

F43nd1r commented 1 year ago

2.0.0 matches bugs based on four criteria:

As I don't have any android apps in production anymore, happy to hear feedback on this and how it could be improved.

The performance issues should be solved with this.