search

wso2 / product-apim

Welcome to the WSO2 API Manager source code! For info on working with the WSO2 API Manager repository and contributing code, click the link below.
http://wso2.github.io/
Apache License 2.0
857 stars 789 forks source link

OutOfMemoryError while many requests get failed. #8450

Open Madhu-NEC opened 4 years ago

Madhu-NEC commented 4 years ago

Hi @yoontaekim, @JKAUSHALYA, @gomathyk, @sherenewso2

While using wso2am 2.1.0 and wso2is 5.3.0 as key manager in our production environment we are facing below issue:

TID: [-1234] [] [2020-05-08 00:00:04,769]  INFO {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator} -  Invalid OAuth Token : Access Token <access token> expired. {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator}
TID: [-1234] [] [2020-05-08 00:00:04,769]  WARN {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} -  API authentication failure due to Invalid Credentials {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
........
TID: [] [] [2020-05-10 17:35:32,655]  INFO {org.apache.synapse.transport.passthru.SourceHandler} -  Writer null when calling informWriterError {org.apache.synapse.transport.passthru.SourceHandler}
TID: [] [] [2020-05-10 17:35:32,655]  WARN {org.apache.synapse.transport.passthru.SourceHandler} -  Connection time out after request is read: http-incoming-10192060 Socket Timeout : 60000 Remote Address : /<IP>:42816 {org.apache.synapse.transport.passthru.SourceHandler}
.........
TID: [] [] [2020-05-10 17:40:35,252]  WARN {org.apache.synapse.transport.passthru.SourceHandler} -  Connection closed by the client after request is read: http-incoming-10192063 Remote Address : null {org.apache.synapse.transport.passthru.SourceHandler}
TID: [] [] [2020-05-10 17:40:35,252]  INFO {org.apache.synapse.transport.passthru.SourceHandler} -  Writer null when calling informWriterError {org.apache.synapse.transport.passthru.SourceHandler}
TID: [] [] [2020-05-10 17:40:35,252]  WARN {org.apache.synapse.transport.passthru.SourceHandler} -  Connection closed by the client after request is read: http-incoming-10192065 Remote Address : null {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2020-05-10 18:21:29,419] ERROR {org.wso2.carbon.registry.core.dataaccess.TransactionManager} -  Failed to start new registry transaction. {org.wso2.carbon.registry.core.dataaccess.TransactionManager}
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935)
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870)
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:902)
       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:892)
       at com.mysql.jdbc.Util.handleNewInstance(Util.java:395)
       at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
       at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
       at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
       at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
       at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
       at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
       at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
       at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
       at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
       at org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.beginTransaction(JDBCLogsDAO.java:325)
       at org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.getLogList(JDBCLogsDAO.java:296)
       at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getLogs(EmbeddedRegistry.java:2374)
       at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getLogs(CacheBackedRegistry.java:485)
       at org.wso2.carbon.registry.core.session.UserRegistry.getLogsInternal(UserRegistry.java:1810)
       at org.wso2.carbon.registry.core.session.UserRegistry.access$3600(UserRegistry.java:61)
       at org.wso2.carbon.registry.core.session.UserRegistry$37.run(UserRegistry.java:1781)
       at org.wso2.carbon.registry.core.session.UserRegistry$37.run(UserRegistry.java:1778)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.wso2.carbon.registry.core.session.UserRegistry.getLogs(UserRegistry.java:1778)
       at org.wso2.carbon.registry.indexing.ResourceSubmitter.submitResource(ResourceSubmitter.java:119)
       at org.wso2.carbon.registry.indexing.ResourceSubmitter.run(ResourceSubmitter.java:76)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
       at com.mysql.jdbc.util.ReadAheadInputStream.<init>(ReadAheadInputStream.java:234)
       at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:310)
       at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2232)
       at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2085)
       at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2069)
       at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
       at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
       at sun.reflect.GeneratedConstructorAccessor193.newInstance(Unknown Source)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
       at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
       at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
       at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
       at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
       at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
       at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
       at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
       at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
       at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
       at org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
       at org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.beginTransaction(JDBCLogsDAO.java:325)
       at org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.getLogList(JDBCLogsDAO.java:296)
       at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getLogs(EmbeddedRegistry.java:2374)
       at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getLogs(CacheBackedRegistry.java:485)
       at org.wso2.carbon.registry.core.session.UserRegistry.getLogsInternal(UserRegistry.java:1810)
       at org.wso2.carbon.registry.core.session.UserRegistry.access$3600(UserRegistry.java:61)
       at org.wso2.carbon.registry.core.session.UserRegistry$37.run(UserRegistry.java:1781)
       at org.wso2.carbon.registry.core.session.UserRegistry$37.run(UserRegistry.java:1778)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.wso2.carbon.registry.core.session.UserRegistry.getLogs(UserRegistry.java:1778)
       at org.wso2.carbon.registry.indexing.ResourceSubmitter.submitResource(ResourceSubmitter.java:119)
       at org.wso2.carbon.registry.indexing.ResourceSubmitter.run(ResourceSubmitter.java:76)

For this, we analyse the heap dumps then we get the following report: memory

Could you please help me in finding the root cause of this issue.

As per the logs mentioned above, initially we faced some Invalid Oauth token error as mentioned in Issue#1868 and then socket time out errors and after that out of memory error. So could you please confirm if this could also be the reason of heap space error?

Madhu-NEC commented 4 years ago

Hi @chathuranga95 , @chanaka3d , @JKAUSHALYA

Could you please help us in resolving this error as we are facing this issue in production environment and it is urgent?