xuxueli / xxl-job

A distributed task scheduling framework.(分布式任务调度平台XXL-JOB)
http://www.xuxueli.com/xxl-job/
GNU General Public License v3.0
27.46k stars 10.86k forks source link

高可用环境下调度中心出现死锁的问题,AdminBizImpl类的registry方法 #1560

Closed qingtianpig closed 4 years ago

qingtianpig commented 4 years ago

2020-04-02 15:50:00.014 INFO 1 — [ool-1-thread-28] c.x.r.r.i.reference.XxlRpcReferenceBean : >>>>>>>>>>> xxl-job, invoke error, address:192.168.6.61:10096, XxlRpcRequestXxlRpcRequest{requestId='ac004806-a519-4fd2-8c31-56e4ec4d347c', createMillisTime=1585813800013, accessToken='', className='com.xxl.job.core.biz.ExecutorBiz', methodName='run', parameterTypes=[class com.xxl.job.core.biz.model.TriggerParam], parameters=[TriggerParam

{jobId=33, executorHandler='clusterHealthCheckJobHandler', executorParams='', executorBlockStrategy='SERIAL_EXECUTION', executorTimeout=0, logId=823362, logDateTim=1585813800010, glueType='BEAN', glueSource='', glueUpdatetime=1553041758000, broadcastIndex=0, broadcastTotal=1} ], version='null'} 2020-04-02 15:50:00.015 ERROR 1 — [ool-1-thread-28] c.x.j.admin.core.trigger.XxlJobTrigger : >>>>>>>>>>> xxl-job trigger error, please check if the executor[192.168.6.61:10096] is running.

com.xxl.rpc.util.XxlRpcException: java.io.EOFException: HttpConnectionOverHTTP@5c56ab93(l:/192.168.6.61:36676 <-> r:/192.168.6.61:10096,closed=false)[HttpChannelOverHTTP@30d38626(exchange=HttpExchange@11ffba3f req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@162be803(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator

{s=START} ],recv=HttpReceiverOverHTTP@7fc8c366(rsp=IDLE,failure=null)[HttpParser

{s=CLOSED,0 of -1} ]]] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:277) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1305) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:182) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:129) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:90) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:174) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748)

at com.xxl.rpc.remoting.invoker.reference.XxlRpcReferenceBean$1.invoke(XxlRpcReferenceBean.java:161) at com.sun.proxy.$Proxy78.run(Unknown Source) at com.xxl.job.admin.core.trigger.XxlJobTrigger.runExecutor(XxlJobTrigger.java:191) at com.xxl.job.admin.core.trigger.XxlJobTrigger.processTrigger(XxlJobTrigger.java:142) at com.xxl.job.admin.core.trigger.XxlJobTrigger.trigger(XxlJobTrigger.java:76) at com.xxl.job.admin.core.thread.JobTriggerPoolHelper$1.run(JobTriggerPoolHelper.java:35) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

2020-04-02 15:50:00.021 ERROR 1 — [SchedulerThread] org.quartz.core.ErrorLogger : An error occurred while scanning for the next triggers to fire.

org.quartz.JobPersistenceException: Couldn't commit jdbc connection. Deadlock: wsrep aborted transaction at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3734) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3851) at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTriggers(JobStoreSupport.java:2802) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:287) Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock: wsrep aborted transaction at sun.reflect.GeneratedConstructorAccessor59.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.Util.getInstance(Util.java:408) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:952) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491) at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1555) at com.zaxxer.hikari.proxy.ConnectionProxy.commit(ConnectionProxy.java:332) at com.zaxxer.hikari.proxy.HikariConnectionProxy.commit(HikariConnectionProxy.java) at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:73) at com.sun.proxy.$Proxy77.commit(Unknown Source) at org.quartz.impl.jdbcjobstore.JobStoreSupport.commitConnection(JobStoreSupport.java:3732) ... 3 common frames omitted

xuxueli commented 4 years ago

你好,从日志看这是基于quartz的低版本,建议升级到最新的Release版本,已经修复该问题。