OpenLiberty / open-liberty

Open Liberty is a highly composable, fast to start, dynamic application server runtime environment
https://openliberty.io
Eclipse Public License 2.0
1.14k stars 587 forks source link

Serviceability - cleanupTransactions generates unnecesary FFDC if XA Connection fails on prepare #27828

Open djmatthews opened 6 months ago

djmatthews commented 6 months ago

XA Connection used in a JTA transaction fails in prepare due to a connection issue.

In this case the J2C code generates 3 FFDCs with stack:

Stack Dump = com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException: DSRA0080E: An exception was received by the Data Store Adapter. See original exception message: Cannot call 'cleanup' on a ManagedConnection while it is still in a transaction.. at com.ibm.ws.rsadapter.impl.WSRdbManagedConnectionImpl.cleanupTransactions(WSRdbManagedConnectionImpl.java:3205) at com.ibm.ws.rsadapter.impl.WSRdbManagedConnectionImpl.cleanup(WSRdbManagedConnectionImpl.java:2737) at com.ibm.ejs.j2c.MCWrapper.cleanup(MCWrapper.java:1549) at com.ibm.ejs.j2c.FreePool.cleanupAndDestroyMCWrapper(FreePool.java:462) at com.ibm.ejs.j2c.FreePool.returnToFreePool(FreePool.java:276) at com.ibm.ejs.j2c.PoolManager.release(PoolManager.java:948) at com.ibm.ejs.j2c.MCWrapper.releaseToPoolManager(MCWrapper.java:2145) at com.ibm.ejs.j2c.MCWrapper.releaseToPoolManager(MCWrapper.java:2133) at com.ibm.ejs.j2c.XATransactionWrapper.afterCompletion(XATransactionWrapper.java:252) at com.ibm.tx.jta.impl.RegisteredSyncs.coreDistributeAfter(RegisteredSyncs.java:302) at com.ibm.tx.jta.impl.RegisteredSyncs.distributeAfter(RegisteredSyncs.java:279) at com.ibm.tx.jta.embeddable.impl.EmbeddableTransactionImpl.distributeAfter(EmbeddableTransactionImpl.java:317) at com.ibm.tx.jta.impl.TransactionImpl.postCompletion(TransactionImpl.java:2529) at com.ibm.tx.jta.impl.TransactionImpl.internalRollback(TransactionImpl.java:1623) at com.ibm.tx.jta.impl.TransactionImpl.coreStage2CommitProcessing(TransactionImpl.java:931) at com.ibm.tx.jta.impl.TransactionImpl.stage2CommitProcessing(TransactionImpl.java:957) at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:818) at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:758) at com.ibm.tx.jta.impl.TranManagerImpl.commit(TranManagerImpl.java:155) at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:112) ...

FFDCs are: Source = com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.cleanupTransactions probeid = 1592 Source = com.ibm.ejs.j2c.MCWrapper.cleanup probeid = 706 and Source = com.ibm.ejs.j2c.poolmanager.FreePool.cleanupAndDestroyMCWrapper probeid = 1140

Ideally there would be no FFDC in this situation - the FFDC shows that the connectionErrorDetected is true. It is probably unnecessary for the cleanup code to be calling cleanupTransactions when connectionErrorDetected==true, nor for that method to be generating FFDC when connectionErrorDetected==true. If FFDC does need to be prodcued for serviceability then the text should not imply that cleanupTransactions should not be called in the current (TRANSACTION_ENDING) state (which makes it looks like a state related bug. FWIW NOTE if the cleanup was false then (as it will be when called from destroy) then cleanupTransactions is a no-op when connectionErrorDetected is called.

djmatthews commented 3 months ago

As discussed with @jmstephensgit ...

The DataStoreAdapterException is raised on line 3214 of the current code in the if(inCleanup) block. It is not a result of calling sqlConn.rollback() which isn't called since currentAutoCommit is true (though connection was used in XA tran). It was agreed that keeping the code path was desriable (ie still throw the exception) but that no FFDC should be produced in certain circumstances.

The DataStoreAdapterException is raised in several cases after an connection error occurred, not just if the failure happens in prepare - also, connection error in commit (including 1PC XA) and rollback and possibly others. In all these cases it was noted from FFDC that: In WSRdbManagedConnectionImpl connectionErrorDetected was true and stateMgr.transtate was TRANSACTION_ENDING (could not determine _mcStale value from FFDC, may be worth adding that to FFDC). In MCWrapper stale and _transactionErrorOccurred were true (though do_not_reuse_mcw was false) FreePool knows that it is going to destroy the wrapper (and also has access to MCWrapper's isStale() method.

3 FFDCs were produced (and should be surpessed in these cases) in WSRdbManagedConnectionImpl.cleanupTransactions, MCWrapper.cleanup and FreePool.cleanupAndDestroyMCWrapper

https://github.com/OpenLiberty/open-liberty/blob/009e78a10a216a16c071fc12685fdb02c7a94191/dev/com.ibm.ws.jdbc/src/com/ibm/ws/rsadapter/impl/WSRdbManagedConnectionImpl.java#L3217 (probeid = 1592) https://github.com/OpenLiberty/open-liberty/blob/integration/dev/com.ibm.ws.jca.cm/src/com/ibm/ejs/j2c/MCWrapper.java#L1570 (probeid = 706) https://github.com/OpenLiberty/open-liberty/blob/009e78a10a216a16c071fc12685fdb02c7a94191/dev/com.ibm.ws.jca.cm/src/com/ibm/ejs/j2c/FreePool.java#L468 (probeid = 1140)

Suggestion is for MCWrapper and FreePool to suppress the FFDC in those objects if MCWrapper.isStale() is true and for WSRdbManagedConnectionImpl to use _mcStale (if confident this is set, or possibly connectionErrorDetected)

@jmstephensgit to assign developer