tytyty123
commented
2 years ago 貌似引发了其他问题，无法连接数据库，检查数据库的连接数，只有十几个，
2022-07-12 10:12:33.412 ERROR 99922 --- [duler_Worker-14] org.quartz.core.ErrorLogger : Unable to notify TriggerListener(s) while firing trigger (Trigger and Job will NOT be fired!). trigger= DEFAULT.AgentPingScheduleJob_trigger job= DEFAULT.AgentPingScheduleJob
org.quartz.SchedulerException: TriggerListener 'LogStoreTriggerListener' threw exception: nested exception is org.apache.ibatis.exceptions.PersistenceException:
Error updating database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms.
The error may exist in com/alibaba/chaosblade/box/dao/mapper/SchedulerJobTriggerLogMapper.java (best guess)
The error may involve com.alibaba.chaosblade.box.dao.mapper.SchedulerJobTriggerLogMapper.insert
The error occurred while executing an update
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms.
at org.quartz.core.QuartzScheduler.notifyTriggerListenersFired(QuartzScheduler.java:1860) ~[quartz-2.3.0.jar!/:na]
at org.quartz.core.JobRunShell.notifyListenersBeginning(JobRunShell.java:296) [quartz-2.3.0.jar!/:na]
at org.quartz.core.JobRunShell.run(JobRunShell.java:173) [quartz-2.3.0.jar!/:na]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.0.jar!/:na]Caused by: org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException:
Error updating database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms.
The error may exist in com/alibaba/chaosblade/box/dao/mapper/SchedulerJobTriggerLogMapper.java (best guess)
The error may involve com.alibaba.chaosblade.box.dao.mapper.SchedulerJobTriggerLogMapper.insert
The error occurred while executing an update
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms.
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:77) ~[mybatis-spring-2.0.1.jar!/:2.0.1]
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446) ~[mybatis-spring-2.0.1.jar!/:2.0.1]
at com.sun.proxy.$Proxy137.insert(Unknown Source) ~[na:na]
at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:278) ~[mybatis-spring-2.0.1.jar!/:2.0.1]
at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:58) ~[mybatis-plus-core-3.1.2.jar!/:3.1.2]
at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:62) ~[mybatis-plus-core-3.1.2.jar!/:3.1.2]
at com.sun.proxy.$Proxy174.insert(Unknown Source) ~[na:na]
at sun.reflect.GeneratedMethodAccessor337.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.13.RELEASE.jar!/:4.3.13.RELEASE]
at com.sun.proxy.$Proxy175.insert(Unknown Source) ~[na:na]
at com.alibaba.chaosblade.box.dao.infrastructure.event.listener.LogStoreTriggerListener.triggerFired(LogStoreTriggerListener.java:52) ~[chaosblade-box-dao-1.0.1.jar!/:na]
at org.quartz.core.QuartzScheduler.notifyTriggerListenersFired(QuartzScheduler.java:1854) ~[quartz-2.3.0.jar!/:na]
... 3 common frames omittedCaused by: org.apache.ibatis.exceptions.PersistenceException:
Error updating database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms.
The error may exist in com/alibaba/chaosblade/box/dao/mapper/SchedulerJobTriggerLogMapper.java (best guess)
The error may involve com.alibaba.chaosblade.box.dao.mapper.SchedulerJobTriggerLogMapper.insert
The error occurred while executing an update
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30) ~[mybatis-3.5.1.jar!/:3.5.1]
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:199) ~[mybatis-3.5.1.jar!/:3.5.1]
at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) ~[mybatis-3.5.1.jar!/:3.5.1]
at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.1.jar!/:2.0.1]
... 20 common frames omittedCaused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms. at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE] at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:82) ~[mybatis-spring-2.0.1.jar!/:2.0.1] at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:68) ~[mybatis-spring-2.0.1.jar!/:2.0.1] at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336) ~[mybatis-3.5.1.jar!/:3.5.1] at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.prepareStatement(MybatisSimpleExecutor.java:93) ~[mybatis-plus-core-3.1.2.jar!/:3.1.2] at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doUpdate(MybatisSimpleExecutor.java:53) ~[mybatis-plus-core-3.1.2.jar!/:3.1.2] at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) ~[mybatis-3.5.1.jar!/:3.5.1] at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) ~[mybatis-3.5.1.jar!/:3.5.1] at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49) ~[mybatis-3.5.1.jar!/:3.5.1] at com.baomidou.mybatisplus.extension.plugins.OptimisticLockerInterceptor.intercept(OptimisticLockerInterceptor.java:97) ~[mybatis-plus-extension-3.1.2.jar!/:3.1.2] at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61) ~[mybatis-3.5.1.jar!/:3.5.1] at com.sun.proxy.$Proxy244.update(Unknown Source) ~[na:na] at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) ~[mybatis-3.5.1.jar!/:3.5.1] ... 25 common frames omitted Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 36919ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548) ~[HikariCP-2.5.1.jar!/:na] at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186) ~[HikariCP-2.5.1.jar!/:na] at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145) ~[HikariCP-2.5.1.jar!/:na] at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:99) ~[HikariCP-2.5.1.jar!/:na] at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE] at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ~[spring-jdbc-4.3.13.RELEASE.jar!/:4.3.13.RELEASE] ... 40 common frames omitted
MandssS
拉的最新分支代码，打包box后，启动运行一段时间，报内存溢出，加了一些jvm参数，输出了一些详细信息
平台只部署了3-5个探针，且只创建了3-5个实验任务（均已停止任务了）
增加的jvm参数如下： -Xms1024m -Xmx4096m -Xss256k -XX:PermSize=128m -XX:MaxPermSize=512m -XX:SurvivorRatio=8 -XX:NewRatio=4 -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintHeapAtGC -XX:+PrintReferenceGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintSafepointStatistics -Xloggc:/app/chaosblade/gc-%t.log -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=50M
2022-07-09 14:43:30.142 ERROR 89512 --- [duler_Worker-39] org.quartz.core.ErrorLogger : Job (DEFAULT.AgentPingScheduleJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. at org.quartz.core.JobRunShell.run(JobRunShell.java:213) ~[quartz-2.3.0.jar!/:na] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.0.jar!/:na] Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) ~[na:1.8.0_161] at java.lang.Thread.start(Thread.java:717) ~[na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) ~[na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) ~[na:1.8.0_161] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[na:1.8.0_161] at com.alibaba.chaosblade.box.scheduler.AgentPingScheduleJob.lambda$execute$1(AgentPingScheduleJob.java:86) ~[classes!/:na] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_161] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[na:1.8.0_161] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_161] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_161] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_161] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_161] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_161] at com.alibaba.chaosblade.box.scheduler.AgentPingScheduleJob.execute(AgentPingScheduleJob.java:87) ~[classes!/:na] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.0.jar!/:na] ... 1 common frames omitted
观察了一会，线程一直在涨，没有被回收，
线程池一直在创建 "pool-1029-thread-1" #3727 prio=5 os_prio=0 tid=0x00007fe84016f560 nid=0x454a waiting on condition [0x00007fe81228a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)
parking to wait for <0x0000000796958ef8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers: