[Bug]: [date 11.20]standalone regression: mo reported context deadline exceeded

heni02 commented 21 hours ago

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch Name

main

Commit ID

76d6b022c943736d9db101d0472f25c103f1924f

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-auto-test/actions/runs/11935391230/job/33266884621 回归18:30:12开始报错context deadline exceeded，之后所有测试都报这个错误

企业微信截图_5d0add89-8d00-47fd-af79-cd0744d057c8

刚客户端登陆失败 [root@mo-srv-129 ~]# mysql -h 127.0.0.1 -udump -P6001 -p111 mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 1815 (HY000): context deadline exceeded

mo log: https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-129%5C%22,%20filename%3D%5C%22%2Fdata3%2Factions-runner%2F_work%2Fmo-auto-test%2Fmo-auto-test%2Fhead%2Fmo-service-76d6b02-20241120-222921.log%5C%22%7D%20%7C%3D%20%60context%20deadline%20exceeded%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221732112965399%22,%22to%22:%221732156795330%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

main standalone regression

Additional information

No response

w-zr commented 21 hours ago

heni02 commented 20 hours ago

goroutine 文件：刚开始报错02:32（utc时间是18:32）时的goroutine： LOG_7c4dccb4-4d3c-41f8-b482-5251dc7a41bf_goroutine_01934ad9-ec75-7f01-8c29-01ff698a49c7.gz

11:12 时的goroutine： LOG_7c4dccb4-4d3c-41f8-b482-5251dc7a41bf_goroutine_01934cb4-9ffe-731b-af86-4cb4c2c0c8d0.gz

daviszhen commented 20 hours ago

就是事务活跃队列占满了。

https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-129%5C%22,%20filename%3D%5C%22%2Fdata3%2Factions-runner%2F_work%2Fmo-auto-test%2Fmo-auto-test%2Fhead%2Fmo-service-76d6b02-20241120-222921.log%5C%22%7D%20%7C%3D%20%60context%20deadline%20exceeded%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221732125600000%22,%22to%22:%221732129199000%22%7D%7D%7D&schemaVersion=1&orgId=1

daviszhen commented 20 hours ago

在 129 上实验了下。

登录不上因为创建事务1分钟超时。

问题是负载都停了依然登录不上。

[root@mo-srv-129 ~]# mysql -h 127.0.0.1 -P 6001 -udump -p111
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1815 (HY000): context deadline exceeded
internal error: createTxnOpUnsafe

20781 goroutine 6404720 [select, 2 minutes]:
20782 github.com/matrixorigin/matrixone/pkg/txn/client.(*waiter).wait(0xc03ba4bcc0, {0x586ed60, 0xc0677be460})
20783         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/timestamp_waiter.go:248 +0x85
20784 github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).waitActive.func2()
20785         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:428 +0x26
20786 github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).doCostAction(0xc05e3f6008, {0x1?, 0xc05e583bd8?, 0x0?}, {0x0?, {0x506f6e5?, 0x0?}}, 0xc05e583c20, 0x8?)
20787         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:1380 +0x395
20788 github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).waitActive(0xc05e3f6008, {0x586ed60, 0xc0677be460})
20789         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:424 +0xfc
20790 github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).doCreateTxn(0xc00110b440, {0x586ed60, 0xc0677be460}, 0xc05e3f6008, {0x0, 0x0, 0x0, {}, {0x0, 0x0, ...}, ...})
20791         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:357 +0x438
20792 github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).New(0xc00110b440, {0x586ed60, 0xc0677be460}, {0x0, 0x0, 0x0, {}, {0x0, 0x0, 0x0}, ...}, ...)
20793         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:286 +0x273
20794 github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createTxnOpUnsafe.func1()
20795         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:422 +0x127
20796 github.com/matrixorigin/matrixone/pkg/frontend.ExecuteFuncWithRecover(0x586ecf0?)
20797         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_buffer.go:870 +0x62
20798 github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createTxnOpUnsafe(0xc062493560, 0xc098a80a00)
20799         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:421 +0xbcf
20800 github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createUnsafe(0xc062493560, 0xc098a80a00)
20801         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:323 +0x22d
20802 github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).Create(0xc062493560, 0xc098a80a00)
20803         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:268 +0xb0
20804 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x59635f8, 0xc04a4c0588}, 0xc05e585238, 0xc098a80a00)
20805         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2536 +0x2af
20806 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x59635f8, 0xc04a4c0588}, 0xc05e585238, 0xc098a80a00)
20807         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2477 +0x118
20808 github.com/matrixorigin/matrixone/pkg/frontend.doComQueryInBack(0xc04a4c0588, 0xc05e585238, 0xc098a80a00, 0xc0677be3f0)
20809         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/back_exec.go:429 +0x123b
20810 github.com/matrixorigin/matrixone/pkg/frontend.(*backExec).Exec(0xce04370498, {0x586ecb8, 0xc4f5a910e0}, {0x505ce68, 0x6})
20811         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/back_exec.go:159 +0x6da
20812 github.com/matrixorigin/matrixone/pkg/frontend.(*Session).AuthenticateUser(0xc379df6008, {0x586ed60, 0xc93786ddc0}, {0xca99967fc0?, 0x0?}, {0x0, 0x0}, {0xc0ccb080f6, 0x14, 0x9e}, ...)
20813         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/session.go:1177 +0x426
20814 github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).authenticateUser(0xc064f90180, {0x586ed60, 0xc93786ddc0}, {0xc0ccb080f6, 0x14, 0x9e})
20815         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_protocol.go:1424 +0x285
20816 github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).Authenticate(0xc064f90180, {0x586ed60, 0xc93786ddc0})
20817         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_protocol.go:1537 +0x178
20818 github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handshake(0xc026c9d540, 0xc02a5526c0)
20819         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:303 +0xcbd
20820 github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn(0xc026c9d540, {0x586ecf0, 0xc026cad400}, {0x58b5318?, 0xc115b89258?})
20821         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:203 +0x358
20822 created by github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept in goroutine 4178
20823         /data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:177 +0x165
20824

daviszhen commented 20 hours ago

io 空闲


[root@mo-srv-129 ~]# iostat 
Linux 5.19.12-1.el8.elrepo.x86_64 (mo-srv-129)  11/21/2024      _x86_64_        (64 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.60    0.00    1.51    0.18    0.00   85.71

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
nvme1n1        1844.83     36389.41     19092.33 102420740412 53736795084
nvme2n1           0.35         3.44        13.68    9668333   38516252
nvme0n1         154.13     10697.48       192.57 30108854070  542001722
sdb              11.93       318.75       243.99  897149478  686741376
sda               4.30       697.27         7.17 1962515246   20177304

cpu 也空闲

top - 12:03:11 up 32 days, 13:50,  7 users,  load average: 0.84, 0.73, 0.78
Tasks: 701 total,   1 running, 700 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.2 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.1 hi,  0.0 si,  0.0 st
MiB Mem : 257392.3 total,  15681.7 free, 122565.5 used, 119145.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 128882.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                      
 361342 github    20   0  646.3g  33.5g  56004 S  46.9  13.3   9009:47 mo-service                                                                                                                                   
2202124 10001     20   0   35.8g   3.4g      0 S   1.7   1.4 860:06.31 sqlservr                                                                                                                                     
      1 root      20   0  239024   7692   4108 S   0.7   0.0 132:17.59 systemd                                                                                                                                      
 213090 systemd+  20   0 2689172 515812      0 S   0.7   0.2 135:42.87 mysqld                                                                                                                                       
 663951 root      20   0   69060   6104   4536 R   0.7   0.0   0:00.09 top                                                                                                                                          
 860211 mysql     20   0   74.7g  72.1g      0 S   0.7  28.7   2265:18 mysqld                                                                                                                                       
1704819 root      20   0 5689384  68764   7064 S   0.7   0.0 354:30.09 promtail                                                                                                                                     
     17 root      20   0       0      0      0 I   0.3   0.0  21:53.15 rcu_preempt

daviszhen commented 19 hours ago

发现有leak

1,第一个leak事务。70f70f4866a9b62a1809b392dd32f716

https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-129%5C%22%7D%20%7C%3D%20%60found%20leak%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

2，事务 70f70f4866a9b62a1809b392dd32f716

报了warn. 从warn的调用占看这个事务应该是创建失败了。既然Rollback了。但是最终还是leak。这个事务的状态应该是错误的。

{"level":"WARN","time":"2024/11/21 02:30:12.190141 +0800","name":"cn-service","caller":"client/client.go:589","msg":"txn closed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1809b392dd32f716","stack":"goroutine 5647901 [running]:
runtime/debug.Stack()
/home/github/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/debug/stack.go:26 +0x5e
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).closeTxn(_, {{0x63, {0x505cbec, 0x6}}, {{0xc041a9dc60, 0x10, 0x10}, 0x5, {0x1809c0a64b4178ee, 0x1, ...}, ...}, ...})
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:591 +0x455
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).triggerEventLocked(...)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator_events.go:74
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).closeLocked(0xc281153810?)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:1268 +0x163
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).Rollback.func2()
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:689 +0x25
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).Rollback(0xc008348008, {0x586ed60, 0xc518c029a0})
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:697 +0x77f
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).doCreateTxn(0xc00110b440, {0x586ed60, 0xc518c029a0}, 0xc008348008, {0x0, 0x0, 0x0, {}, {0x0, 0x0, ...}, ...})
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:358 +0x473
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).New(0xc00110b440, {0x586ed60, 0xc518c029a0}, {0x0, 0x0, 0x0, {}, {0x0, 0x0, 0x0}, ...}, ...)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:286 +0x273
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createTxnOpUnsafe.func1()
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:422 +0x127
github.com/matrixorigin/matrixone/pkg/frontend.ExecuteFuncWithRecover(0x586ecf0?)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_buffer.go:870 +0x62
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createTxnOpUnsafe(0xc0765e6d80, 0xc1bd660a00)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:421 +0xbcf
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createUnsafe(0xc0765e6d80, 0xc1bd660a00)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:323 +0x22d
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).Create(0xc0765e6d80, 0xc1bd660a00)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:268 +0xb0
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x5963308, 0xc516277208}, 0x0, 0xc1bd660a00)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2536 +0x2af
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x5963308, 0xc516277208}, 0x0, 0xc1bd660a00)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2477 +0x118
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse(0xc516277208, 0xc1bd660a00)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2440 +0x3d5
github.com/matrixorigin/matrixone/pkg/frontend.doComQuery(0xc516277208, 0xc1bd660a00, 0xc518c02930)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:3074 +0x1d74
github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest(0xc516277208, 0xc1bd660a00, 0xc281155b88)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:3151 +0x78c
github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest(0xd22dc26d80, 0xc281155b88)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/routine.go:298 +0x61d
github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler(0xc026cfe180, 0xc1f4f0c6c0, {0xc0576d2d20, 0xed, 0xed})
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/routine_manager.go:385 +0x327
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleRequest(0xc026c9d540, 0xc1f4f0c6c0)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:516 +0x1eb
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage(0xc026c9d540, {0x586ecf0, 0xc026cad400}, 0xc1f4f0c6c0)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:484 +0x94
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleLoop(0xc026c9d540?, {0x586ecf0?, 0xc026cad400?}, 0xc00b434a80?)
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:212 +0x2f
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn(0xc026c9d540, {0x586ecf0, 0xc026cad400}, {0x58b5318?, 0xc3f31bb420?})
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:208 +0x4a6
created by github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept in goroutine 4178
/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:177 +0x165
"}

https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-129%5C%22%7D%20%7C%3D%20%6070f70f4866a9b62a1809b392dd32f716%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

daviszhen commented 19 hours ago

看了前10个leak的事务。现象跟上面的一样。

https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22GYP%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-129%5C%22%7D%20%7C%3D%20%60found%20leak%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

daviszhen commented 19 hours ago

还有一个 restarttxn的事务也是类似


{"level":"WARN","time":"2024/11/21 02:30:12.215225 +0800","name":"cn-service","caller":"client/client.go:589","msg":"txn closed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1809b392dd32f720","stack":"goroutine 5647226 [running]:
runtime/debug.Stack()
\t/home/github/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/debug/stack.go:26 +0x5e
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).closeTxn(_, {{0x63, {0x505cbec, 0x6}}, {{0xc3ce66ed90, 0x10, 0x10}, 0x5, {0x1809c0a64b4178ee, 0x1, ...}, ...}, ...})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:591 +0x455
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).triggerEventLocked(...)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator_events.go:74
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).closeLocked(0xc06d910000?)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:1268 +0x163
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).Rollback.func2()
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:689 +0x25
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnOperator).Rollback(0xc208438008, {0x586ed60, 0xc049661d50})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/operator.go:697 +0x77f
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).doCreateTxn(0xc00110b440, {0x586ed60, 0xc049661d50}, 0xc208438008, {0x0, 0x0, 0x0, {}, {0x0, 0x0, ...}, ...})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:358 +0x473
github.com/matrixorigin/matrixone/pkg/txn/client.(*txnClient).RestartTxn(0xc00110b440, {0x586ed60, 0xc049661d50}, {0x593ba70?, 0xc208438008}, {0x0, 0x0, 0x0, {}, {0x0, ...}, ...}, ...)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/txn/client/client.go:304 +0x294
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createTxnOpUnsafe.func2()
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:430 +0x143
github.com/matrixorigin/matrixone/pkg/frontend.ExecuteFuncWithRecover(0xd2343b1fa0?)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_buffer.go:870 +0x62
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createTxnOpUnsafe(0xc0e407dc20, 0xc2b9745040)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:429 +0xa4f
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).createUnsafe(0xc0e407dc20, 0xc2b9745040)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:323 +0x22d
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).Create(0xc0e407dc20, 0xc2b9745040)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/txn.go:268 +0xb0
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x59635f8, 0xc1008c1348}, 0xc06d911690, 0xc2b9745040)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2536 +0x2af
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x59635f8, 0xc1008c1348}, 0xc06d911690, 0xc2b9745040)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2477 +0x118
github.com/matrixorigin/matrixone/pkg/frontend.doComQueryInBack(0xc1008c1348, 0xc06d911690, 0xc2b9745040, 0xc049661ce0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/back_exec.go:429 +0x123b
github.com/matrixorigin/matrixone/pkg/frontend.(*backExec).Exec(0xc0397f6150, {0x586ecb8, 0xc4cf7bc690}, {0x505ce68, 0x6})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/back_exec.go:159 +0x6da
github.com/matrixorigin/matrixone/pkg/frontend.determineUserHasPrivilegeSet({0x586ecb8, 0xc4cf7bc690}, 0xc51f425208, 0xc0ed1046e0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/authenticate.go:6107 +0x26a
github.com/matrixorigin/matrixone/pkg/frontend.authenticateUserCanExecuteStatementWithObjectTypeDatabaseAndTable({0x586ecb8, 0xc4cf7bc690}, 0xc51f425208, {0x5892328?, 0xc0d53ea510?}, 0xc27a3b1300)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/authenticate.go:6704 +0xb1
github.com/matrixorigin/matrixone/pkg/frontend.authenticateCanExecuteStatementAndPlan({0x586ecb8, 0xc4cf7bc690}, 0xc51f425208, {0x5892328, 0xc0d53ea510}, 0xc27a3b1300)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2221 +0x105
github.com/matrixorigin/matrixone/pkg/frontend.buildPlan({0x586ecb8, 0xc52a3ac5d0}, {0x5963308, 0xc51f425208}, {0x593a040, 0xc637779c80}, {0x5892328, 0xc0d53ea510})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:1995 +0xaf1
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnComputationWrapper).Compile(0xc20ffacc80, {0x4715b80?, 0xc1cc4ad2c0}, 0xc1063394b8)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/computation_wrapper.go:196 +0x505
github.com/matrixorigin/matrixone/pkg/frontend.executeStmt(0xc51f425208, 0xc1cc4ad2c0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2779 +0xb2e
github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt({0x5963308, 0xc51f425208}, 0x0, 0xc1cc4ad2c0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2673 +0x2ef
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt({0x5963308, 0xc51f425208}, 0x0, 0xc1cc4ad2c0, {0x593ba70, 0xc074c2c008})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2642 +0x35c
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x5963308, 0xc51f425208}, 0x0, 0xc1cc4ad2c0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2586 +0x567
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x5963308, 0xc51f425208}, 0x0, 0xc1cc4ad2c0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2477 +0x118
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse(0xc51f425208, 0xc1cc4ad2c0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:2440 +0x3d5
github.com/matrixorigin/matrixone/pkg/frontend.doComQuery(0xc51f425208, 0xc1cc4ad2c0, 0xc03ac48850)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:3074 +0x1d74
github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest(0xc51f425208, 0xc1cc4ad2c0, 0xc2c4f13b88)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/mysql_cmd_executor.go:3151 +0x78c
github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest(0xc637779c00, 0xc2c4f13b88)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/routine.go:298 +0x61d
github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler(0xc026cfe180, 0xc1ae9a37a0, {0xc05a92a5a0, 0xed, 0xed})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/routine_manager.go:385 +0x327
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleRequest(0xc026c9d540, 0xc1ae9a37a0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:516 +0x1eb
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage(0xc026c9d540, {0x586ecf0, 0xc026cad400}, 0xc1ae9a37a0)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:484 +0x94
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleLoop(0xc026c9d540?, {0x586ecf0?, 0xc026cad400?}, 0xc00b434a80?)
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:212 +0x2f
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn(0xc026c9d540, {0x586ecf0, 0xc026cad400}, {0x58b5318?, 0xc49c5de590?})
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:208 +0x4a6
created by github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept in goroutine 4178
\t/data3/actions-runner/_work/mo-auto-test/mo-auto-test/head/pkg/frontend/server.go:177 +0x165
"}

matrixorigin / matrixone