search

michaelklishin / quartz-mongodb

A MongoDB-based store for the Quartz scheduler. This fork strives to be as feature complete as possible. Originally by MuleSoft.
Other
249 stars 203 forks source link

Unstable mongodb cause scheduler to be hanged? #174

Closed jamie0828 closed 4 years ago

jamie0828 commented 5 years ago

In our production system some times the quartz schedule stop working, the schedule is active but no trigger is fired. I found the error log below and it may be the cause. After the scheduler job is done, the quartz try to notify the quartz store and change something (maybe the status or what ever). Once the mangodb is unstable the notify can not be execute and cause the job in uncomplete status and cause problem. Any solution?

[11:09:29:889] [ERROR] - org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:585) - Error while executing the Runnable: com.mongodb.MongoQueryException: Query failed with error code 202 and error message 'Couldn't get a connection within the time limit' on server 10.172.48.25:27017 at com.mongodb.operation.FindOperation$1.call(FindOperation.java:707) ~[mongodb-driver-core-3.9.1.jar:?] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:696) ~[mongodb-driver-core-3.9.1.jar:?] at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:462) ~[mongodb-driver-core-3.9.1.jar:?] at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:406) ~[mongodb-driver-core-3.9.1.jar:?] at com.mongodb.operation.FindOperation.execute(FindOperation.java:696) ~[mongodb-driver-core-3.9.1.jar:?] at com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~[mongodb-driver-core-3.9.1.jar:?] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179) ~[mongodb-driver-3.9.1.jar:?] at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:199) ~[mongodb-driver-3.9.1.jar:?] at com.novemberain.quartz.mongodb.dao.JobDao.getById(JobDao.java:61) ~[quartz-mongodb-2.1.0.jar:?] at com.novemberain.quartz.mongodb.trigger.TriggerConverter.toTriggerWithOptionalJob(TriggerConverter.java:99) ~[quartz-mongodb-2.1.0.jar:?] at com.novemberain.quartz.mongodb.trigger.TriggerConverter.toTrigger(TriggerConverter.java:69) ~[quartz-mongodb-2.1.0.jar:?] at com.novemberain.quartz.mongodb.dao.TriggerDao.getTrigger(TriggerDao.java:103) ~[quartz-mongodb-2.1.0.jar:?] at com.novemberain.quartz.mongodb.JobCompleteHandler.process(JobCompleteHandler.java:69) ~[quartz-mongodb-2.1.0.jar:?] at com.novemberain.quartz.mongodb.JobCompleteHandler.jobComplete(JobCompleteHandler.java:54) ~[quartz-mongodb-2.1.0.jar:?] at com.novemberain.quartz.mongodb.MongoDBJobStore.triggeredJobComplete(MongoDBJobStore.java:437) ~[quartz-mongodb-2.1.0.jar:?] at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1804) ~[quartz-2.2.3.jar:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:269) ~[quartz-2.2.3.jar:?] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.3.jar:?]

michaelklishin commented 4 years ago

Slow job store operations can potentially affect Quartz but I doubt it. In any case, there isn't much this job store can do. I can't comment on how realistic this hypothesis is, please take it to a Quartz discussion forum.