matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 277 forks source link

[Bug]: [1102 main tke regression] tpcc 500-1000 report 'cannot commit a orphan transaction'. #19750

Open Ariznawlll opened 1 month ago

Ariznawlll commented 1 month ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

7faf76e882cbef0bc7e1365824428de4c6406ec7

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job url: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11630838813/job/32398275649

image

tpcc 500-1000执行期间log:https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22uoa%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-7faf76e88-20241101%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221730488184000%22,%22to%22:%221730490087000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

trigger daily regression workflow

Additional information

No response

sukki37 commented 4 weeks ago

https://grafana.ci.matrixorigin.cn/goto/IkioBCWNR?orgId=1 During the error period, the CPU usage of the CN is not high.

zhangxu19830126 commented 3 weeks ago

https://grafana.ci.matrixorigin.cn/goto/yQwUm7GHg?orgId=1

大量的io timeout。几乎所有的组件都在i/o time

zhangxu19830126 commented 3 weeks ago

goroutine的调度的延迟非常大,有100ms

image
zhangxu19830126 commented 3 weeks ago

goroutine 数量也很大

image
zhangxu19830126 commented 3 weeks ago

gc 占用cpu也很大

image
fengttt commented 3 weeks ago

19965

19966

19967

daviszhen commented 2 weeks ago

https://github.com/matrixorigin/ci-test/actions/runs/11810386696/job/32903395595

zhangxu19830126 commented 2 weeks ago

暂时无法解决,等田博士的那些issue被fix了再看看情况

zhangxu19830126 commented 1 week ago

暂时无法解决,等田博士的那些 issue 被 fix 了再看看情况

zhangxu19830126 commented 6 days ago

暂时无法解决,等田博士的那些 issue 被 fix 了再看看情况

zhangxu19830126 commented 13 hours ago

暂时无法解决,等田博士的那些 issue 被 fix 了再看看情况