matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 276 forks source link

[Bug]: [date 9.6]tke regression:tpch 1T 4cn Q9 test mo hung #18621

Closed heni02 closed 1 month ago

heni02 commented 2 months ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

0eba06817

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/10740237043/job/29813556249

企业微信截图_8b248493-1a71-4352-afd9-2c2abd2b962b

mo log long running: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22qFM%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-0eba06817-20240906%5C%22%7D%20%7C%3D%20%60long%20running%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221725633837541%22,%22to%22:%221725720237541%22%7D%7D%7D&schemaVersion=1&orgId=1

leakcheck profile文件: CN_39343065-3130-6638-3463-343731646166_leakcheck_routine_0191cb61-c050-786d-ba4e-243b1cd6d622.gz

Expected Behavior

No response

Steps to Reproduce

tke tpch1T 4cn test

Additional information

No response

heni02 commented 2 months ago

date 9.8 tpch1T 4cn load时候 mo hung住

企业微信截图_9c05028a-9d40-4db3-9559-0c36a688579f

leakcheck_routine文件: CN_36633637-3461-6239-3334-666165336435_leakcheck_routine_0191d5fb-5387-7c75-b0f7-949fa71aad4a.gz

badboynt1 commented 2 months ago

update async task表 hang住,复现太困难,需要针对这张表的负载情况专门写个test

badboynt1 commented 2 months ago

https://github.com/matrixorigin/matrixone/pull/18495 等这个pr合并后再观察一下

badboynt1 commented 2 months ago

今天daily调整用时后没有hang住。 说明需要跑够12个小时以上可能是触发条件之一

badboynt1 commented 2 months ago

这个pr修复了一个delete pipeline中的bug,可能跟这个hang有关。 等合并进去以后再跑跑看

badboynt1 commented 1 month ago

https://github.com/matrixorigin/matrixone/pull/18495 等这个pr合并后再观察一下

badboynt1 commented 1 month ago

https://github.com/matrixorigin/matrixone/pull/18495 这个pr合并后没有再hang过

badboynt1 commented 1 month ago

改了ddl之后已经连续3次没有hang住了 再观察一下

badboynt1 commented 1 month ago

最近的四次都没有再复现了。

heni02 commented 1 month ago

confirm,closed