search

apache / incubator-seata

:fire: Seata is an easy-to-use, high-performance, open source distributed transaction solution.
https://seata.apache.org/
Apache License 2.0
25.23k stars 8.76k forks source link

XA模式资源悬挂问题 #4073

Closed lcmvs closed 2 years ago

lcmvs commented 2 years ago

Ⅰ. Issue Description

XA模式资源悬挂问题

Ⅱ. Describe what happened

XA模式下，TC日志分布式事务回滚成功，实际上数据库中的分支事务未回滚，导致资源永远无法处理。

Ⅲ. Describe what you expected to happen

TC日志记录的分布式事务回滚成功，数据库中的分支事务也应该回滚成功。

Ⅳ. How to reproduce it (as minimally and precisely as possible)

TM注册一个分布式事务到TC

RM注册一个分支事务到TC

但是RM还没有执行xa end，就因为超时，TC发起执行了xa rollback

这时候因为隔离性，xa rollback会抛出一个异常XAException.XAER_NOTA，无效的xid

TC认为rollback成功，删除掉branch_table的数据

这时候RM的xa end执行完毕，数据库出现不一致的bug

流程

TM RM TC

注册分布式事务

​ 分布式事务注册成功

​ xa start 123;

​ 分布式事务超时回滚

​ xa rollback 123;

​ 分布式事务回滚成功

​ xa end 123;

Ⅴ. Anything else we need to know?

问题代码

我的理解就是下面的代码导致的这个bug

但是这里涉及到另外一个问题：https://github.com/seata/seata/issues/2971

直接删除掉这段代码又会导致TC无限重试扽问题

private BranchStatus finishBranch(boolean committed, BranchType branchType, String xid, long branchId, String resourceId,
                                   String applicationData) throws TransactionException {
    XAXid xaBranchXid = XAXidBuilder.build(xid, branchId);
    Resource resource = dataSourceCache.get(resourceId);
    if (resource instanceof AbstractDataSourceProxyXA) {
        try (ConnectionProxyXA connectionProxyXA = ((AbstractDataSourceProxyXA)resource).getConnectionForXAFinish(xaBranchXid)) {
            if (committed) {
                connectionProxyXA.xaCommit(xid, branchId, applicationData);
                LOGGER.info(xaBranchXid + " was committed.");
                return BranchStatus.PhaseTwo_Committed;
            } else {
                connectionProxyXA.xaRollback(xid, branchId, applicationData);
                LOGGER.info(xaBranchXid + " was rollbacked");
                return BranchStatus.PhaseTwo_Rollbacked;
            }
        } catch (XAException | SQLException sqle) {
            //我的理解这个问题就是因为这一段代码导致的 
            if (sqle instanceof XAException) {
                if (((XAException)sqle).errorCode == XAException.XAER_NOTA) {
                    if (committed) {
                        return BranchStatus.PhaseTwo_Committed;
                    } else {
                        return BranchStatus.PhaseTwo_Rollbacked;
                    }
                }
            }
            if (committed) {
                LOGGER.info(xaBranchXid + " commit failed since " + sqle.getMessage(), sqle);
                // FIXME: case of PhaseTwo_CommitFailed_Unretryable
                return BranchStatus.PhaseTwo_CommitFailed_Retryable;
            } else {
                LOGGER.info(xaBranchXid + " rollback failed since " + sqle.getMessage(), sqle);
                // FIXME: case of PhaseTwo_RollbackFailed_Unretryable
                return BranchStatus.PhaseTwo_RollbackFailed_Retryable;
            }
        }
    } else {
        LOGGER.error("Unknown Resource for XA resource " + resourceId + " " + resource);
        if (committed) {
            return BranchStatus.PhaseTwo_CommitFailed_Unretryable;
        } else {
            return BranchStatus.PhaseTwo_RollbackFailed_Unretryable;
        }
    }
}

问题重现

XA RECOVER;
formatID gtrid_length bqual_length data
9752 39 20 192.168.97.113:8091:1459353340120858789-1459353340120858793

TC日志

15:23:56.058  INFO --- [     batchLoggerPrint_1_1] i.s.c.r.p.server.BatchLogHandler         : SeataMergeMessage timeout=60000,transactionName=fsp-create-order
,clientIp:192.168.97.113,vgroup:my-test-tx-group
15:23:56.107  INFO --- [verHandlerThread_1_15_500] i.s.s.coordinator.DefaultCoordinator     : Begin new global transaction applicationId: order-server,transactionServiceGroup: my-test-tx-group, transactionName: fsp-create-order,timeout:60000,xid:192.168.97.113:8091:1459353340120858789
15:23:56.309  INFO --- [     batchLoggerPrint_1_1] i.s.c.r.p.server.BatchLogHandler         : SeataMergeMessage xid=192.168.97.113:8091:1459353340120858789,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-order,lockKey=null
,clientIp:192.168.97.113,vgroup:my-test-tx-group
15:23:56.363  INFO --- [verHandlerThread_1_16_500] i.seata.server.coordinator.AbstractCore  : Register branch successfully, xid = 192.168.97.113:8091:1459353340120858789, branchId = 1459353340120858791, resourceId = jdbc:mysql://119.29.246.230:3306/seat-order ,lockKeys = null
15:23:56.749  INFO --- [     batchLoggerPrint_1_1] i.s.c.r.p.server.BatchLogHandler         : SeataMergeMessage xid=192.168.97.113:8091:1459353340120858789,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,lockKey=null
,clientIp:192.168.97.113,vgroup:my-test-tx-group
15:23:56.797  INFO --- [verHandlerThread_1_17_500] i.seata.server.coordinator.AbstractCore  : Register branch successfully, xid = 192.168.97.113:8091:1459353340120858789, branchId = 1459353340120858793, resourceId = jdbc:mysql://119.29.246.230:3306/seat-storage ,lockKeys = null
15:24:01.776  INFO --- [     batchLoggerPrint_1_1] i.s.c.r.p.server.BatchLogHandler         : SeataMergeMessage xid=192.168.97.113:8091:1459353340120858789,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,lockKey=null
,clientIp:192.168.97.113,vgroup:my-test-tx-group
15:24:01.829  INFO --- [verHandlerThread_1_18_500] i.seata.server.coordinator.AbstractCore  : Register branch successfully, xid = 192.168.97.113:8091:1459353340120858789, branchId = 1459353340120858801, resourceId = jdbc:mysql://119.29.246.230:3306/seat-storage ,lockKeys = null
15:24:06.511  INFO --- [     batchLoggerPrint_1_1] i.s.c.r.p.server.BatchLogHandler         : SeataMergeMessage xid=192.168.97.113:8091:1459353340120858789,extraData=null
,clientIp:192.168.97.113,vgroup:my-test-tx-group
15:24:07.114  INFO --- [verHandlerThread_1_19_500] io.seata.server.coordinator.DefaultCore  : Rollback branch transaction successfully, xid = 192.168.97.113:8091:1459353340120858789 branchId = 1459353340120858801
15:24:07.254  INFO --- [verHandlerThread_1_19_500] io.seata.server.coordinator.DefaultCore  : Rollback branch transaction successfully, xid = 192.168.97.113:8091:1459353340120858789 branchId = 1459353340120858793
15:24:07.419  INFO --- [verHandlerThread_1_19_500] io.seata.server.coordinator.DefaultCore  : Rollback branch transaction successfully, xid = 192.168.97.113:8091:1459353340120858789 branchId = 1459353340120858791
15:24:07.606  INFO --- [verHandlerThread_1_19_500] io.seata.server.coordinator.DefaultCore  : Rollback global transaction successfully, xid = 192.168.97.113:8091:1459353340120858789.
15:24:53.087  INFO --- [     batchLoggerPrint_1_1] i.s.c.r.p.server.BatchLogHandler         : SeataMergeMessage xid=192.168.97.113:8091:1459353340120858789,branchId=1459353340120858801,resourceId=null,status=PhaseOne_Failed,applicationData=null
,clientIp:192.168.97.113,vgroup:my-test-tx-group
15:24:53.111 ERROR --- [verHandlerThread_1_23_500] i.s.c.e.AbstractExceptionHandler         : Catch TransactionException while do RPC, request: xid=192.168.97.113:8091:1459353340120858789,branchId=1459353340120858801,resourceId=null,status=PhaseOne_Failed,applicationData=null
==>
io.seata.core.exception.GlobalTransactionException: Could not found global transaction xid = 192.168.97.113:8091:1459353340120858789, may be has finished.

RM日志

2021-10-13 15:24:06.576  INFO 18684 --- [h_RMROLE_1_2_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:1459353340120858789,branchId=1459353340120858801,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-13 15:24:06.577  INFO 18684 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:1459353340120858789 1459353340120858801 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-13 15:24:06.808  INFO 18684 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-13 15:24:07.116  INFO 18684 --- [h_RMROLE_1_3_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:1459353340120858789,branchId=1459353340120858793,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-13 15:24:07.116  INFO 18684 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:1459353340120858789 1459353340120858793 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-13 15:24:07.144  INFO 18684 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked

Ⅵ. Environment:

funky-eyes commented 2 years ago

XAER_NOTA 无法区分xid是真的回滚了还是假的，因为比如tc下发个rm，rm回滚完毕，此时tc挂了，或者rm挂了，tc再次下发的时候也是XAER_NOTA这个状态

lcmvs commented 2 years ago

XAER_NOTA 无法区分xid是真的回滚了还是假的，因为比如tc下发个rm，rm回滚完毕，此时tc挂了，或者rm挂了，tc再次下发的时候也是XAER_NOTA这个状态

是的，但是我感觉资源悬挂比无限重试，问题更大。 还有这样是不是只能让dba手动处理了？ 有更好的解决方式吗？

funky-eyes commented 2 years ago

XAER_NOTA 无法区分xid是真的回滚了还是假的，因为比如tc下发个rm，rm回滚完毕，此时tc挂了，或者rm挂了，tc再次下发的时候也是XAER_NOTA这个状态

是的，但是我感觉资源悬挂比无限重试，问题更大。 还有这样是不是只能让dba手动处理了？ 有更好的解决方式吗？

可以在这一起讨论下优化方案，目前先让dba手动处理下吧

funky-eyes commented 2 years ago

我目前想到的方案，因为在prepare之后才hold连接，此时可以改成start就保持连接，这样下发的时候就可以拿到这个正在被其他线程使用的connection进行直接rollback这个xid，后续的end和prepare就会使业务线程直接抛出异常，你怎么看？

funky-eyes commented 2 years ago

4074 麻烦有空看下当前pr的设计是否可以解决该问题,欢迎继续讨论

lcmvs commented 2 years ago

4074 麻烦有空看下当前pr的设计是否可以解决该问题,欢迎继续讨论

就是让rollback和未完成一阶段的xa事务处于同一个连接，这样就不会有隔离性的问题了，是吧，我的理解是可以解决这个问题的。

我用你的pr尝试重现这个问题，的确不会再发生了。 我用另外一个事务，锁住storage资源 然后让seata开启一个分布式事务 如果是之前的流程，会直接超时回滚成功，出现资源悬挂 改成你的pr以后，因为使用了同一个连接，rollback也会被锁住，直到抛出锁超时异常

2021-10-14 10:48:54.259  INFO 11964 --- [h_RMROLE_1_2_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634445,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:48:54.262  INFO 11964 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634445 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:24.700  INFO 11964 --- [h_RMROLE_1_3_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634445,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:49:24.700  INFO 11964 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634445 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:35.895  INFO 11964 --- [nio-8182-exec-1] i.s.rm.datasource.xa.ConnectionProxyXA   : 192.168.97.113:8091:837856899279634433-837856899279634438 was rollbacked
2021-10-14 10:49:35.926 ERROR 11964 --- [nio-8182-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may exist in file [D:\lcm\my-test\seata\seata-eureka-sample-lcm\springcloud-eureka-feign-mybatis-seata-xa\storage-server\target\classes\mapper\StorageMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select * from storage where product_id = ? for update
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause

com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:948) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2082) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3298) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:498) ~[druid-1.1.10.jar:1.1.10]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.lambda$execute$2(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ExecuteTemplateXA.execute(ExecuteTemplateXA.java:47) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.execute(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.5.0.jar:3.5.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at com.sun.proxy.$Proxy102.selectOne(Unknown Source) ~[na:na]
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58) ~[mybatis-3.5.0.jar:3.5.0]
    at com.sun.proxy.$Proxy103.selectForUpdate(Unknown Source) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at com.sun.proxy.$Proxy105.selectForUpdate(Unknown Source) ~[na:na]
    at io.seata.sample.service.StorageServiceImpl.decrease(StorageServiceImpl.java:36) ~[classes/:na]
    at io.seata.sample.service.StorageServiceImpl$$FastClassBySpringCGLIB$$e940a87d.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at io.seata.sample.service.StorageServiceImpl$$EnhancerBySpringCGLIB$$7cec0344.decrease(<generated>) ~[classes/:na]
    at io.seata.sample.controller.StorageController.decrease(StorageController.java:28) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.10.jar:1.1.10]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

2021-10-14 10:49:40.749  INFO 11964 --- [h_RMROLE_1_2_24] i.s.rm.datasource.xa.ResourceManagerXA   : 192.168.97.113:8091:837856899279634433-837856899279634445 rollback failed since XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state

com.mysql.jdbc.jdbc2.optional.MysqlXAException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:581) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:566) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.rollback(MysqlXAConnection.java:411) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:122) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:113) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ResourceManagerXA.finishBranch(ResourceManagerXA.java:74) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ResourceManagerXA.branchRollback(ResourceManagerXA.java:59) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler.doBranchRollback(AbstractRMHandler.java:125) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:67) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:124) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler.handle(AbstractRMHandler.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.DefaultRMHandler.handle(DefaultRMHandler.java:68) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.protocol.transaction.BranchRollbackRequest.handle(BranchRollbackRequest.java:35) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler.onRequest(AbstractRMHandler.java:150) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.handleBranchRollback(RmBranchRollbackProcessor.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.process(RmBranchRollbackProcessor.java:58) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.rpc.netty.AbstractNettyRemoting.lambda$processMessage$2(AbstractNettyRemoting.java:279) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
Caused by: java.sql.SQLException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:959) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3806) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2470) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2617) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:840) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:740) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:560) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    ... 20 common frames omitted

2021-10-14 10:49:40.749  INFO 11964 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_RollbackFailed_Retryable
2021-10-14 10:49:40.804  INFO 11964 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-14 10:49:40.846  INFO 11964 --- [nio-8182-exec-2] i.s.rm.datasource.xa.ConnectionProxyXA   : null was rollbacked
2021-10-14 10:49:40.847 ERROR 11964 --- [nio-8182-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may exist in file [D:\lcm\my-test\seata\seata-eureka-sample-lcm\springcloud-eureka-feign-mybatis-seata-xa\storage-server\target\classes\mapper\StorageMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select * from storage where product_id = ? for update
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause

com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:948) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2082) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3298) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:498) ~[druid-1.1.10.jar:1.1.10]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.lambda$execute$2(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ExecuteTemplateXA.execute(ExecuteTemplateXA.java:47) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.execute(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.5.0.jar:3.5.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at com.sun.proxy.$Proxy102.selectOne(Unknown Source) ~[na:na]
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58) ~[mybatis-3.5.0.jar:3.5.0]
    at com.sun.proxy.$Proxy103.selectForUpdate(Unknown Source) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at com.sun.proxy.$Proxy105.selectForUpdate(Unknown Source) ~[na:na]
    at io.seata.sample.service.StorageServiceImpl.decrease(StorageServiceImpl.java:36) ~[classes/:na]
    at io.seata.sample.service.StorageServiceImpl$$FastClassBySpringCGLIB$$e940a87d.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at io.seata.sample.service.StorageServiceImpl$$EnhancerBySpringCGLIB$$7cec0344.decrease(<generated>) ~[classes/:na]
    at io.seata.sample.controller.StorageController.decrease(StorageController.java:28) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.10.jar:1.1.10]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

2021-10-14 10:49:40.981  INFO 11964 --- [h_RMROLE_1_4_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634438,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:49:40.982  INFO 11964 --- [h_RMROLE_1_4_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634438 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:41.010  INFO 11964 --- [h_RMROLE_1_4_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-14 10:51:16.462  INFO 11964 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2021-10-14 10:51:20.602  INFO 11964 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
funky-eyes commented 2 years ago

4074 麻烦有空看下当前pr的设计是否可以解决该问题,欢迎继续讨论

就是让rollback和未完成一阶段的xa事务处于同一个连接，这样就不会有隔离性的问题了，是吧，我的理解是可以解决这个问题的。

我用你的pr尝试重现这个问题，的确不会再发生了。 我用另外一个事务，锁住storage资源 然后让seata开启一个分布式事务 如果是之前的流程，会直接超时回滚成功，出现资源悬挂 改成你的pr以后，因为使用了同一个连接，rollback也会被锁住，直到抛出锁超时异常

2021-10-14 10:48:54.259  INFO 11964 --- [h_RMROLE_1_2_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634445,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:48:54.262  INFO 11964 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634445 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:24.700  INFO 11964 --- [h_RMROLE_1_3_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634445,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:49:24.700  INFO 11964 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634445 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:35.895  INFO 11964 --- [nio-8182-exec-1] i.s.rm.datasource.xa.ConnectionProxyXA   : 192.168.97.113:8091:837856899279634433-837856899279634438 was rollbacked
2021-10-14 10:49:35.926 ERROR 11964 --- [nio-8182-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may exist in file [D:\lcm\my-test\seata\seata-eureka-sample-lcm\springcloud-eureka-feign-mybatis-seata-xa\storage-server\target\classes\mapper\StorageMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select * from storage where product_id = ? for update
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause

com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:948) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2082) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3298) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:498) ~[druid-1.1.10.jar:1.1.10]
  at io.seata.rm.datasource.xa.PreparedStatementProxyXA.lambda$execute$2(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.ExecuteTemplateXA.execute(ExecuteTemplateXA.java:47) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.PreparedStatementProxyXA.execute(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.5.0.jar:3.5.0]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
  at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.0.jar:2.0.0]
  at com.sun.proxy.$Proxy102.selectOne(Unknown Source) ~[na:na]
  at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-2.0.0.jar:2.0.0]
  at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58) ~[mybatis-3.5.0.jar:3.5.0]
  at com.sun.proxy.$Proxy103.selectForUpdate(Unknown Source) ~[na:na]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at com.sun.proxy.$Proxy105.selectForUpdate(Unknown Source) ~[na:na]
  at io.seata.sample.service.StorageServiceImpl.decrease(StorageServiceImpl.java:36) ~[classes/:na]
  at io.seata.sample.service.StorageServiceImpl$$FastClassBySpringCGLIB$$e940a87d.invoke(<generated>) ~[classes/:na]
  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at io.seata.sample.service.StorageServiceImpl$$EnhancerBySpringCGLIB$$7cec0344.decrease(<generated>) ~[classes/:na]
  at io.seata.sample.controller.StorageController.decrease(StorageController.java:28) ~[classes/:na]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.10.jar:1.1.10]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

2021-10-14 10:49:40.749  INFO 11964 --- [h_RMROLE_1_2_24] i.s.rm.datasource.xa.ResourceManagerXA   : 192.168.97.113:8091:837856899279634433-837856899279634445 rollback failed since XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state

com.mysql.jdbc.jdbc2.optional.MysqlXAException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
  at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:581) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:566) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.rollback(MysqlXAConnection.java:411) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:122) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:113) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.ResourceManagerXA.finishBranch(ResourceManagerXA.java:74) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.ResourceManagerXA.branchRollback(ResourceManagerXA.java:59) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.AbstractRMHandler.doBranchRollback(AbstractRMHandler.java:125) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:67) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:124) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.AbstractRMHandler.handle(AbstractRMHandler.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.DefaultRMHandler.handle(DefaultRMHandler.java:68) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.core.protocol.transaction.BranchRollbackRequest.handle(BranchRollbackRequest.java:35) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.AbstractRMHandler.onRequest(AbstractRMHandler.java:150) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.handleBranchRollback(RmBranchRollbackProcessor.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.process(RmBranchRollbackProcessor.java:58) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.core.rpc.netty.AbstractNettyRemoting.lambda$processMessage$2(AbstractNettyRemoting.java:279) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
  at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
Caused by: java.sql.SQLException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
  at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:959) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3806) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2470) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2617) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:840) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:740) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:560) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  ... 20 common frames omitted

2021-10-14 10:49:40.749  INFO 11964 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_RollbackFailed_Retryable
2021-10-14 10:49:40.804  INFO 11964 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-14 10:49:40.846  INFO 11964 --- [nio-8182-exec-2] i.s.rm.datasource.xa.ConnectionProxyXA   : null was rollbacked
2021-10-14 10:49:40.847 ERROR 11964 --- [nio-8182-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may exist in file [D:\lcm\my-test\seata\seata-eureka-sample-lcm\springcloud-eureka-feign-mybatis-seata-xa\storage-server\target\classes\mapper\StorageMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select * from storage where product_id = ? for update
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause

com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
  at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:948) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2082) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3298) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.37.jar:5.1.37]
  at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.1.10.jar:1.1.10]
  at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:498) ~[druid-1.1.10.jar:1.1.10]
  at io.seata.rm.datasource.xa.PreparedStatementProxyXA.lambda$execute$2(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.ExecuteTemplateXA.execute(ExecuteTemplateXA.java:47) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at io.seata.rm.datasource.xa.PreparedStatementProxyXA.execute(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
  at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.5.0.jar:3.5.0]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
  at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.0.jar:2.0.0]
  at com.sun.proxy.$Proxy102.selectOne(Unknown Source) ~[na:na]
  at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-2.0.0.jar:2.0.0]
  at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84) ~[mybatis-3.5.0.jar:3.5.0]
  at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58) ~[mybatis-3.5.0.jar:3.5.0]
  at com.sun.proxy.$Proxy103.selectForUpdate(Unknown Source) ~[na:na]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at com.sun.proxy.$Proxy105.selectForUpdate(Unknown Source) ~[na:na]
  at io.seata.sample.service.StorageServiceImpl.decrease(StorageServiceImpl.java:36) ~[classes/:na]
  at io.seata.sample.service.StorageServiceImpl$$FastClassBySpringCGLIB$$e940a87d.invoke(<generated>) ~[classes/:na]
  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at io.seata.sample.service.StorageServiceImpl$$EnhancerBySpringCGLIB$$7cec0344.decrease(<generated>) ~[classes/:na]
  at io.seata.sample.controller.StorageController.decrease(StorageController.java:28) ~[classes/:na]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.10.jar:1.1.10]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.24.jar:9.0.24]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

2021-10-14 10:49:40.981  INFO 11964 --- [h_RMROLE_1_4_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634438,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:49:40.982  INFO 11964 --- [h_RMROLE_1_4_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634438 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:41.010  INFO 11964 --- [h_RMROLE_1_4_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-14 10:51:16.462  INFO 11964 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2021-10-14 10:51:20.602  INFO 11964 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration

不是很明白你说的rollback被锁住，能发下你的流程吗？

lcmvs commented 2 years ago

4074 麻烦有空看下当前pr的设计是否可以解决该问题,欢迎继续讨论

就是让rollback和未完成一阶段的xa事务处于同一个连接，这样就不会有隔离性的问题了，是吧，我的理解是可以解决这个问题的。 我用你的pr尝试重现这个问题，的确不会再发生了。 我用另外一个事务，锁住storage资源 然后让seata开启一个分布式事务 如果是之前的流程，会直接超时回滚成功，出现资源悬挂 改成你的pr以后，因为使用了同一个连接，rollback也会被锁住，直到抛出锁超时异常

2021-10-14 10:48:54.259  INFO 11964 --- [h_RMROLE_1_2_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634445,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:48:54.262  INFO 11964 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634445 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:24.700  INFO 11964 --- [h_RMROLE_1_3_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634445,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:49:24.700  INFO 11964 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634445 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:35.895  INFO 11964 --- [nio-8182-exec-1] i.s.rm.datasource.xa.ConnectionProxyXA   : 192.168.97.113:8091:837856899279634433-837856899279634438 was rollbacked
2021-10-14 10:49:35.926 ERROR 11964 --- [nio-8182-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may exist in file [D:\lcm\my-test\seata\seata-eureka-sample-lcm\springcloud-eureka-feign-mybatis-seata-xa\storage-server\target\classes\mapper\StorageMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select * from storage where product_id = ? for update
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause

com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:948) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2082) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3298) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:498) ~[druid-1.1.10.jar:1.1.10]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.lambda$execute$2(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ExecuteTemplateXA.execute(ExecuteTemplateXA.java:47) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.execute(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.5.0.jar:3.5.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at com.sun.proxy.$Proxy102.selectOne(Unknown Source) ~[na:na]
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58) ~[mybatis-3.5.0.jar:3.5.0]
    at com.sun.proxy.$Proxy103.selectForUpdate(Unknown Source) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at com.sun.proxy.$Proxy105.selectForUpdate(Unknown Source) ~[na:na]
    at io.seata.sample.service.StorageServiceImpl.decrease(StorageServiceImpl.java:36) ~[classes/:na]
    at io.seata.sample.service.StorageServiceImpl$$FastClassBySpringCGLIB$$e940a87d.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at io.seata.sample.service.StorageServiceImpl$$EnhancerBySpringCGLIB$$7cec0344.decrease(<generated>) ~[classes/:na]
    at io.seata.sample.controller.StorageController.decrease(StorageController.java:28) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.10.jar:1.1.10]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

2021-10-14 10:49:40.749  INFO 11964 --- [h_RMROLE_1_2_24] i.s.rm.datasource.xa.ResourceManagerXA   : 192.168.97.113:8091:837856899279634433-837856899279634445 rollback failed since XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state

com.mysql.jdbc.jdbc2.optional.MysqlXAException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:581) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:566) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.rollback(MysqlXAConnection.java:411) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:122) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ConnectionProxyXA.xaRollback(ConnectionProxyXA.java:113) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ResourceManagerXA.finishBranch(ResourceManagerXA.java:74) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ResourceManagerXA.branchRollback(ResourceManagerXA.java:59) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler.doBranchRollback(AbstractRMHandler.java:125) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:67) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler$2.execute(AbstractRMHandler.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:124) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler.handle(AbstractRMHandler.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.DefaultRMHandler.handle(DefaultRMHandler.java:68) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.protocol.transaction.BranchRollbackRequest.handle(BranchRollbackRequest.java:35) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.AbstractRMHandler.onRequest(AbstractRMHandler.java:150) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.handleBranchRollback(RmBranchRollbackProcessor.java:63) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.rpc.processor.client.RmBranchRollbackProcessor.process(RmBranchRollbackProcessor.java:58) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.core.rpc.netty.AbstractNettyRemoting.lambda$processMessage$2(AbstractNettyRemoting.java:279) [seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_171]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]
Caused by: java.sql.SQLException: XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:959) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3806) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2470) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2617) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:840) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:740) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:560) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    ... 20 common frames omitted

2021-10-14 10:49:40.749  INFO 11964 --- [h_RMROLE_1_2_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_RollbackFailed_Retryable
2021-10-14 10:49:40.804  INFO 11964 --- [h_RMROLE_1_3_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-14 10:49:40.846  INFO 11964 --- [nio-8182-exec-2] i.s.rm.datasource.xa.ConnectionProxyXA   : null was rollbacked
2021-10-14 10:49:40.847 ERROR 11964 --- [nio-8182-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException: 
### Error querying database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may exist in file [D:\lcm\my-test\seata\seata-eureka-sample-lcm\springcloud-eureka-feign-mybatis-seata-xa\storage-server\target\classes\mapper\StorageMapper.xml]
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: select * from storage where product_id = ? for update
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause

com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.Util.getInstance(Util.java:387) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:948) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2082) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3298) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1192) ~[mysql-connector-java-5.1.37.jar:5.1.37]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167) ~[druid-1.1.10.jar:1.1.10]
    at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:498) ~[druid-1.1.10.jar:1.1.10]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.lambda$execute$2(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.ExecuteTemplateXA.execute(ExecuteTemplateXA.java:47) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at io.seata.rm.datasource.xa.PreparedStatementProxyXA.execute(PreparedStatementProxyXA.java:66) ~[seata-all-1.4.2-lcm-SNAPSHOT.jar:1.4.2-lcm-SNAPSHOT]
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) ~[mybatis-3.5.0.jar:3.5.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at com.sun.proxy.$Proxy102.selectOne(Unknown Source) ~[na:na]
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:166) ~[mybatis-spring-2.0.0.jar:2.0.0]
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:84) ~[mybatis-3.5.0.jar:3.5.0]
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58) ~[mybatis-3.5.0.jar:3.5.0]
    at com.sun.proxy.$Proxy103.selectForUpdate(Unknown Source) ~[na:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at com.sun.proxy.$Proxy105.selectForUpdate(Unknown Source) ~[na:na]
    at io.seata.sample.service.StorageServiceImpl.decrease(StorageServiceImpl.java:36) ~[classes/:na]
    at io.seata.sample.service.StorageServiceImpl$$FastClassBySpringCGLIB$$e940a87d.invoke(<generated>) ~[classes/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at io.seata.sample.service.StorageServiceImpl$$EnhancerBySpringCGLIB$$7cec0344.decrease(<generated>) ~[classes/:na]
    at io.seata.sample.controller.StorageController.decrease(StorageController.java:28) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123) ~[druid-1.1.10.jar:1.1.10]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) ~[spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.24.jar:9.0.24]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

2021-10-14 10:49:40.981  INFO 11964 --- [h_RMROLE_1_4_24] i.s.c.r.p.c.RmBranchRollbackProcessor    : rm handle branch rollback process:xid=192.168.97.113:8091:837856899279634433,branchId=837856899279634438,branchType=XA,resourceId=jdbc:mysql://119.29.246.230:3306/seat-storage,applicationData=null
2021-10-14 10:49:40.982  INFO 11964 --- [h_RMROLE_1_4_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacking: 192.168.97.113:8091:837856899279634433 837856899279634438 jdbc:mysql://119.29.246.230:3306/seat-storage
2021-10-14 10:49:41.010  INFO 11964 --- [h_RMROLE_1_4_24] io.seata.rm.AbstractRMHandler            : Branch Rollbacked result: PhaseTwo_Rollbacked
2021-10-14 10:51:16.462  INFO 11964 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2021-10-14 10:51:20.602  INFO 11964 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration

不是很明白你说的rollback被锁住，能发下你的流程吗？

其他事务：BEGIN; 其他事务：SELECT FROM storage WHERE id=1 FOR UPDATE; seata：xa start 123； seata：SELECT FROM storage WHERE id=1 FOR UPDATE; seata：xa rollback 123;

funky-eyes commented 2 years ago

另一个事务应该是一个新的connection，不应该存在共用上一个事务的connection的问题才对 目前的逻辑就是registry后立马hold连接，此时如果再registry到start中间出现rollback，那么本地会插入一个状态，强行让start抛出异常，走rollback逻辑，我在想这个start应该是要允许通过还是不通过，如果start不允许通过，那么rollback将出现异常，因为rollback会end事务再rollback，走的都是xa的rollback逻辑，我应该改一下start的检测逻辑，先start，再检测是否需要中断事务

funky-eyes commented 2 years ago

你能写成事务A此时的行为，跟事务B的行为， 时间段内在各自在做什么行为吗

funky-eyes commented 2 years ago

seata 在setautocommit的时候会直接进行start，你如何做到先select for update 在进行xa start的？

lcmvs commented 2 years ago

seata 在setautocommit的时候会直接进行start，你如何做到先select for update 在进行xa start的？

事务A是我在seata外部开启的一个事务 事务B是seata的一个分支事务 事务A提前锁住了storage的资源，然后一直没有提交

1.事务A：BEGIN; 2.事务A：SELECT FROM storage WHERE id=1 FOR UPDATE; 3.事务B：xa start 123； 4.事务B：SELECT FROM storage WHERE id=1 FOR UPDATE;（被事务A锁住） 5.事务B：xa rollback 123;（TC超时回滚）

funky-eyes commented 2 years ago

原来如此，这样我就明白了，你看下现在是不是不会发生悬挂了，我又提交了个代码，让xaconnection先start再进行检测是否中断，这样在start后可以正常rollback，否者在框架层调用connection.rollbakc执行end的时候可能会报错

funky-eyes commented 2 years ago

XAER_RMFAIL 应该是由于没有start的时候直接进行了rollback导致的，我现在改成了先start再检测，由于XAER_RMFAIL状态tc会进行重试，所以等到start后，要么是本地提前终止，要么是rollback 成功

funky-eyes commented 2 years ago

帮忙再多验证一些悬挂的场景，比如end后 prepare前， end前 start后，start前这样，谢谢

lcmvs commented 2 years ago

XAER_RMFAIL 应该是由于没有start的时候直接进行了rollback导致的，我现在改成了先start再检测，由于XAER_RMFAIL状态tc会进行重试，所以等到start后，要么是本地提前终止，要么是rollback 成功

不是的，XAER_RMFAIL是因为start之后没有end，就直接执行了rollback。 image

funky-eyes commented 2 years ago

我知道，所以我说XAER_RMFAIL的时候tc是会重试下发的rm的，而如果是XAER_NOTA，会在本地缓存状态，提前终止业务线程的connection操作

funky-eyes commented 2 years ago

如果是XAER_RMFAIL tc就会重试，rm那边迟早会走到end这步，所以影响不大

lcmvs commented 2 years ago

帮忙再多验证一些悬挂的场景，比如end后 prepare前， end前 start后，start前这样，谢谢

end前 start后 就是我上面模拟的场景，现在进行rollback不会再出现资源悬挂的问题了。 end后 prepare前 和 start前 我也尝试用debug的方式来模拟，也没有出现资源悬挂的问题。

funky-eyes commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

lcmvs commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

你好，发现了一个问题。

order，account，storage三个服务，resource_id是相同的，也就是相同数据库，不同应用的场景下，假设开启了一个全局事务，然后order和tm的channel断了，tm会选择一个resource_id相同的服务，作为替代，比如account，然后又会发生xaer_note的异常，最终会发生资源悬挂的问题，而且不仅仅是回滚，提交也会发生。

Choose {} on the same IP[{}] as alternative of {}

lcmvs commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

你好，发现了一个问题。

order，account，storage三个服务，resource_id是相同的，也就是相同数据库，不同应用的场景下，假设开启了一个全局事务，然后order和tm的channel断了，tm会选择一个resource_id相同的服务，作为替代，比如account，然后又会发生xaer_note的异常，最终会发生资源悬挂的问题，而且不仅仅是回滚，提交也会发生。

Choose {} on the same IP[{}] as alternative of {}

lcmvs commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

` if (resultChannel == null) { resultChannel = tryOtherApp(applicationIdMap, targetApplicationId);

        if (resultChannel == null) {
            if (LOGGER.isInfoEnabled()) {
                LOGGER.info("No channel is available for resource[{}] as alternative of {}", resourceId, clientId);
            }
        } else {
            if (LOGGER.isInfoEnabled()) {
                LOGGER.info("Choose {} on the same resource[{}] as alternative of {}", resultChannel, resourceId, clientId);
            }
        }
    }`
funky-eyes commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

你好，发现了一个问题。

order，account，storage三个服务，resource_id是相同的，也就是相同数据库，不同应用的场景下，假设开启了一个全局事务，然后order和tm的channel断了，tm会选择一个resource_id相同的服务，作为替代，比如account，然后又会发生xaer_note的异常，最终会发生资源悬挂的问题，而且不仅仅是回滚，提交也会发生。

Choose {} on the same IP[{}] as alternative of {}

提个新的issue，一起聊一下吧

funky-eyes commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

你好，发现了一个问题。

order，account，storage三个服务，resource_id是相同的，也就是相同数据库，不同应用的场景下，假设开启了一个全局事务，然后order和tm的channel断了，tm会选择一个resource_id相同的服务，作为替代，比如account，然后又会发生xaer_note的异常，最终会发生资源悬挂的问题，而且不仅仅是回滚，提交也会发生。

Choose {} on the same IP[{}] as alternative of {}

我估计要把原来本地锁和本地缓存悬挂事务的global状态的方案换为分布式锁和存储来应对这种情况，这样就保证了当order只是网络断开时，这样的话，在order的commit/rollback和account的commit/rollback变成串行，如果是account先进行rollback，直接往分布式存储（比如redis）里塞branchstatus，此时虽然回滚失败，但是order那边走到commit时取出缓存，直接回滚当前事务避免被悬挂即可。跟原定方案一样，本地缓存和锁升级为全局的即可，你怎么看？

lcmvs commented 2 years ago

好的，非常感谢，有问题欢迎继续反馈

你好，发现了一个问题。 order，account，storage三个服务，resource_id是相同的，也就是相同数据库，不同应用的场景下，假设开启了一个全局事务，然后order和tm的channel断了，tm会选择一个resource_id相同的服务，作为替代，比如account，然后又会发生xaer_note的异常，最终会发生资源悬挂的问题，而且不仅仅是回滚，提交也会发生。 Choose {} on the same IP[{}] as alternative of {}

提个新的issue，一起聊一下吧

提了新的issue https://github.com/seata/seata/issues/4138