matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.77k stars 274 forks source link

[Bug]: [date 9.20]tke regression: tpch 1T Q18 caused cn oom #18901

Open heni02 opened 4 days ago

heni02 commented 4 days ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

06cd080dd

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/10961185574/job/30439548651

企业微信截图_7b1ac62d-c4f2-4583-bdd9-48012eaa2b64 企业微信截图_2392169a-c184-4b14-9fe6-6a107bc477f0

oom前 2024-09-21 01:02:17 heap: CN_38623834-6239-6461-3033-393364333139_heap_01921062-67d7-7cc9-9ade-58663ad1d455.gz oom前 2024-09-21 01:02:47 heap: CN_38623834-6239-6461-3033-393364333139_heap_01921062-dd07-78d1-b549-7cd73f76aadf.gz

oom前 2024-09-21 01:02:18 goroutine: CN_38623834-6239-6461-3033-393364333139_goroutine_01921062-695c-766e-af78-592d090e76d2.gz oomq前2024-09-21 01:02:47 goroutine: CN_38623834-6239-6461-3033-393364333139_goroutine_01921062-de09-7986-add0-62dfaf3352fa.gz

grafana profile: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22B_U%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-06cd080dd-20240920%5C%22,pod%3D%5C%22nightly-regression-dis-tp-cn-dkh87%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22memory:alloc_objects:count:space:bytes%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221726851710000%22,%22to%22:%221726851787000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke tpch 1T 3cn test

Additional information

No response

badboynt1 commented 4 days ago

已经通过二分,确定是https://github.com/matrixorigin/matrixone/pull/18852 这个pr引入。 先revert掉再慢慢找原因。

badboynt1 commented 2 days ago

fixed