search

AtlasOfLivingAustralia / logger-service

Atlas event logging
https://logger.ala.org.au
1 stars 8 forks source link

Any data view randomly throws error page. #35

Open nickdos opened 3 years ago

nickdos commented 3 years ago

As noted in #30 but also observed for https://logger-test.ala.org.au/service/logger/sources and similar calls. Often occurs when there has been a long period of inactivity and after a few failed calls, comes good again by itself. Page displays "AN error occurred" message. Logs show these errors:

Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 78,957,557 milliseconds ago. The last packet sent successfully to the server was 78,957,557 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
        at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
        at com.mysql.cj.protocol.a.NativeProtocol.clearInputStream(NativeProtocol.java:815)
        at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:630)
        at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:948)
        at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:894)
        at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1073)
        at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2051)
        ... 52 common frames omitted
Caused by: java.io.IOException: Socket is closed
        at com.mysql.cj.protocol.AbstractSocketConnection.getMysqlInput(AbstractSocketConnection.java:72)
        at com.mysql.cj.protocol.a.NativeProtocol.clearInputStream(NativeProtocol.java:811)
        ... 57 common frames omitted

Seems to have started when Tomcat 9 and MySQL version 8 was updated.

nickdos commented 3 years ago

Possible fixes:

https://stackoverflow.com/questions/26422632/jdbc-connection-timeout-cannot-reconnect https://stackoverflow.com/questions/45114889/the-last-packet-successfully-received-from-the-server-was-43417-seconds-ago

Rita-C commented 3 years ago 
2021-08-06 09:46:53.663 ERROR --- [0.1-8080-exec-2] o.g.web.errors.GrailsExceptionResolver   : SQLNonTransientConnectionException occurred when processing request: [GET] /admin/logEventType
Communications link failure during rollback(). Transaction resolution unknown.. Stacktrace follows:

org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: Unable to rollback against JDBC Connection
        at org.springframework.orm.hibernate5.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:611)
        at org.grails.orm.hibernate.GrailsHibernateTransactionManager.doRollback(GrailsHibernateTransactionManager.groovy:83)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:857)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:834)
        at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:164)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
        at grails.gorm.transactions.GrailsTransactionTemplate.execute(GrailsTransactionTemplate.groovy:91)
        at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)
        at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)
        at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:54)
        at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)
        at org.jasig.cas.client.util.HttpServletRequestWrapperFilter.doFilter(HttpServletRequestWrapperFilter.java:71)
        at org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:236)
        at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:188)
        at org.jasig.cas.client.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:168)
        at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:188)
        at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97)
        at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
        at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:128)
        at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
        at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:103)
        at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:121)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: Unable to rollback against JDBC Connection
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:120)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:250)
        at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:89)
        at org.springframework.orm.hibernate5.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:608)
        ... 30 common frames omitted
Caused by: java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1853)
        at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:370)
        at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:244)
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:115)
        ... 33 common frames omitted