Open stuartpullinger opened 6 years ago
Instructions to edit the domain.xml file (found at payara41/glassfish/domains/domain1/config/domain.xml) are here.
I think this would be equivalent to the asadmin command:
asadmin set server.ejb-container.ejb-timer-service.property.reschedule-failed-timer="true"
but I haven't tested it.
The Expunged timers seem to occur when there is a problem with the connection to the database. Database queries hang which leads to scheduled actions failing which leads to the timers being expunged.
One guard against this happening could be to set a timeout for database queries. It can be done in the persistence.xml file and in the code. We currently have Payara version 4.1.2-173 in production which ships with EclipseLink 2.6.4. The reference documentation for setting the timeout in Eclipselink 2.6 is here.
More info (using a more recent version of the spec) is here. The JPA spec states that this new query hint is optional but the documentation for our implementation (EclipseLink) suggest it is supported in version 2.7. The spec states that the value should be milliseconds but the EclipseLink docs say they default to seconds so, if we upgrade Payara and start using the newer javax.persistence.query.timeout
hint, care should be taken to also define the eclipselink.jdbc.timeout.unit
hint to make clear the units.
Glassfish behaves differently to JBoss and Geronimo in expunging timers after an application exception rather than just logging the exception. Expunging the timers in Glassfish can only be fixed by redeploying the application.
We should add the 'reschedule-failed-timer' option to the setup-glassfish.py script to prevent the Timers from being expunged.
Some references: https://stackoverflow.com/a/18303655 https://github.com/javaee/ejb-spec/issues/111 - requesting clarity in the spec https://github.com/javaee/glassfish/issues/20749 - reporting issue to Glassfish. Doesn't look like it will be addressed. https://bitbucket.org/obfischer/glassfish-timerproblem/src - example application to produce the error