search

whiskyzhu / myschedule

Automatically exported from code.google.com/p/myschedule
0 stars 0 forks source link

ConcurrentModificationException #73

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
This is for the latest version.

What steps will reproduce the problem?
1. Using latest version with JobHistory plugin
2. Create my own job.
3. Try and run job manually. (Exception is thrown when I hit "run it now")

What is the expected output? Job should run.

What do you see instead?  Here is the logging stacktrace
2012-02-21 03:34:15,657 ERROR 
[org.quartz.core.ErrorLogger.schedulerError(QuartzScheduler.java:2360)] - Error 
notifying listeners of trigger misfire.
org.quartz.SchedulerException: TriggerListener 'MyJobHistoryPlugin' threw 
exception: Failed to close DB connection. [See nested exception: 
myschedule.quartz.extra.QuartzRuntimeException: Failed to close DB connection.]
        at org.quartz.core.QuartzScheduler.notifyTriggerListenersMisfired(QuartzScheduler.java:1858)
        at org.quartz.core.SchedulerSignalerImpl.notifyTriggerListenersMisfired(SchedulerSignalerImpl.java:74)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doUpdateOfMisfiredTrigger(JobStoreSupport.java:1017)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:975)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3187)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3946)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3967)
Caused by: myschedule.quartz.extra.QuartzRuntimeException: Failed to close DB 
connection.
        at myschedule.quartz.extra.JdbcSchedulerHistoryPlugin.withConn(JdbcSchedulerHistoryPlugin.java:196)
        at myschedule.quartz.extra.JdbcSchedulerHistoryPlugin.insertHistory(JdbcSchedulerHistoryPlugin.java:169)
        at myschedule.quartz.extra.JdbcSchedulerHistoryPlugin.access$900(JdbcSchedulerHistoryPlugin.java:95)
        at myschedule.quartz.extra.JdbcSchedulerHistoryPlugin$HistoryTriggerListener.triggerMisfired(JdbcSchedulerHistoryPlugin.java:572)
        at org.quartz.core.QuartzScheduler.notifyTriggerListenersMisfired(QuartzScheduler.java:1856)
        ... 6 more
Caused by: java.sql.SQLException: An SQLException was provoked by the following 
failure: java.util.ConcurrentModificationException
        at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
        at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
        at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:62)
        at com.mchange.v2.c3p0.impl.NewProxyConnection.close(NewProxyConnection.java:1271)
        at myschedule.quartz.extra.JdbcSchedulerHistoryPlugin.withConn(JdbcSchedulerHistoryPlugin.java:193)
        ... 10 more
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:806)
        at java.util.HashMap$KeyIterator.next(HashMap.java:841)
        at com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinAll(GooGooStatementCache.java:286)
        at com.mchange.v2.c3p0.impl.NewPooledConnection.checkinAllCachedStatements(NewPooledConnection.java:589)
        at com.mchange.v2.c3p0.impl.NewPooledConnection.markClosedProxyConnection(NewPooledConnection.java:298)
        at com.mchange.v2.c3p0.impl.NewProxyConnection.close(NewProxyConnection.java:1246)
        ... 11 more

Please use labels and text to provide additional information.

Original issue reported on code.google.com by tomans...@gmail.com on 21 Feb 2012 at 3:37

GoogleCodeExporter commented 9 years ago 
Hum... interesting. I haven't seen this before. The stacktrace indicate that 
this looks like a BUG in the c3p0 library! The MySchedule is simply using the 
borrowed connection object and close() as it should be.

Can you show me what's your quartz.properties looks like? Try to bump up the 
maxConnections with +1 and see if your problem still occur. 

If problem still occurs, you might want to switch to other DB conn pool and see 
if it helps? (Eg: try BoneCP from http://jolbox.com/) But you will have to 
write an org.quartz.utils.ConnectionProvider implementations though, but which 
is very simple to do.

Original comment by saltnlight5 on 24 Feb 2012 at 4:10

GoogleCodeExporter commented 9 years ago 
I have reported the issue on c3p0 project.
https://sourceforge.net/tracker/?func=detail&aid=3492734&group_id=25357&atid=383
690

Original comment by saltnlight5 on 24 Feb 2012 at 4:23

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I will try and up the maxConnections and see what happens.

config properties as requested - 

# Main Quartz configuration
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = CMVClustSched
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = 
org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.dataSource = quartzDataSource
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 5

# JobStore: JDBC jobStoreTX
org.quartz.dataSource.quartzDataSource.driver = com.mysql.jdbc.Driver
org.quartz.dataSource.quartzDataSource.URL = 
jdbc:mysql://localhost:3306/cmvscheduler
org.quartz.dataSource.quartzDataSource.user = cmv
org.quartz.dataSource.quartzDataSource.password =
# Recommend threadPool size + 3
org.quartz.dataSource.quartzDataSource.maxConnections = 8

# MySchedule scheduler service parameters (These are not used by quartz itself, 
but for the webapp only.)
myschedule.schedulerService.autoInit = true
myschedule.schedulerService.autoStart = true
myschedule.schedulerService.waitForJobsToComplete = true

# Jdbc Scheduler History Plugin
org.quartz.plugin.MyJobHistoryPlugin.class = 
myschedule.quartz.extra.JdbcSchedulerHistoryPlugin
org.quartz.plugin.MyJobHistoryPlugin.insertSql = INSERT INTO 
qrtz_scheduler_history VALUES(?,?,?,?,?,?,?,?,?,?,?)
org.quartz.plugin.MyJobHistoryPlugin.querySql = SELECT * FROM 
qrtz_scheduler_history ORDER BY event_time DESC
org.quartz.plugin.MyJobHistoryPlugin.deleteSql = DELETE qrtz_scheduler_history 
WHERE event_time < ?
org.quartz.plugin.MyJobHistoryPlugin.deleteIntervalInSecs = 604800
org.quartz.plugin.MyJobHistoryPlugin.dataSourceName = quartzDataSource
org.quartz.plugin.MyJobHistoryPlugin.schedulerContextKey = 
JdbcSchedulerHistoryPlugin.Instance

Original comment by tomans...@gmail.com on 24 Feb 2012 at 4:44

GoogleCodeExporter commented 9 years ago 
Well, the "# Recommend threadPool size + 3" is for quartz use only. Now that we 
do use extra for the plugin, so yes do increase by one more.

But before you do that, I just saw an update from c3p0 ticket that they 
recommend us upgrade to latest version ("c3p0-0.9.1.2"). Do you think you can 
give that a try? 

The one comes with Quartz is "c3p0-0.9.1.1", which was release back in 2007!

Original comment by saltnlight5 on 24 Feb 2012 at 10:23

GoogleCodeExporter commented 9 years ago 
Just update the jar in the war I already have?

Original comment by tomans...@gmail.com on 24 Feb 2012 at 10:29

GoogleCodeExporter commented 9 years ago 
I have updated to the latest c3p0 and the problem still persists. =(

I will look into it further and see if I can find anything.

Original comment by tomans...@gmail.com on 24 Feb 2012 at 10:38

GoogleCodeExporter commented 9 years ago 
BTW, I have upped the maxConnections to 35 so that should be enough.  Still
the problem persists!

Original comment by tomans...@gmail.com on 24 Feb 2012 at 10:39

GoogleCodeExporter commented 9 years ago 
Hum... that's no good. When I have some time, I can review further on the 
plugin code to see if we doing things with connection correctly. But if you 
continue to get the same exception stack, then the problem is still on c3p0 
though.

If you are willing to try, I do have a BoneCP ConnectionProvider implemented. 
Here is an example of quartz.properties on how to use it:
http://code.google.com/p/myschedule/source/browse/bonecp-quartz/src/test/resourc
es/deng/quartz/bonecp/quartz.properties?repo=experiment

For convenience, I will attach the two jars that you will need.

Original comment by saltnlight5 on 25 Feb 2012 at 3:52

Attachments:

GoogleCodeExporter commented 9 years ago 
I think I might have solved this problem but not sure.  I have been reading 
that the mysql global system property "wait_timeout" might create this kind of 
issue.  The default value is 28800.  Mine was 15.  

I changed it back to 28800 and things seemed to have stabilized quite a bit.  
Which to me suggests my value was way too low which was causing the connection 
to be closed by MySQL and then it would not be available when needed by 
MyScheduler which was trying to access a closed connection.

I'll keep monitoring and post back if I see any changes.

Original comment by tomans...@gmail.com on 26 Feb 2012 at 9:43

GoogleCodeExporter commented 9 years ago 
Cool, thanks for the update.

Original comment by saltnlight5 on 26 Feb 2012 at 9:55

GoogleCodeExporter commented 9 years ago 
close as wont-fix

Original comment by saltnlight5 on 21 Mar 2012 at 3:38