search

matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.78k stars 276 forks source link

[Bug]: restore account + tpch 1t query, 3 cn oom #19399

Open tom-csf opened 1 week ago

tom-csf commented 1 week ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

b848638952bd5a0dad5498863136e256caa14d07

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

3 cn oom https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22QAQ%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%27mo-search-nightly-b84863895-20241016%27,pod%3D%27nightly-regression-dis-tp-cn-xrmjd%27%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22memory:alloc_space:bytes:space:bytes%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221729141200000%22,%22to%22:%221729143000000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

1、创建的account 级别的pitr 
2、用脚本循环进行restore account pitr 数据规模20w条
3、tpch 1t 性能测试同步进行

Additional information

企业微信截图_2918b8cf-41f2-4155-81d7-632df7e73110 企业微信截图_0738c021-8c3b-4c0b-8e16-808fe815a7d4 企业微信截图_0c39bf0c-d544-4a3a-af93-9071d5483d66 image
triump2020 commented 1 week ago

可能跟 snapshot 对应的 partition state 很多有关，导致内存占用过大, 没有gc 掉.

triump2020 commented 1 week ago

在处理其他S-1

triump2020 commented 6 days ago

需要gc 快照对应的partition state .

volgariver6 commented 1 day ago

本地复现步骤：

  1. MO 启动，通过dump用户创建新的租户：

    create account acc1 admin_name 'root' identified by '111';

  2. acc1:root 用户户登录，建库，建表：

    create database db01;
use db01
create table tab2 (c1 int, c2 varchar(100));

  3. 创建pitr：

    ./mo_br pitr create --host 127.0.0.1 --port 6001 --user "acc1#root" --password 111 --pname pitr11 --level account --account acc1 --rangevalue 2 --rangeunit d

  4. insert 数据，中间看一下时间：

    insert into tab2 values (100, 'aaaaaaaaaaaaaaa');
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
select now();
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
insert into tab2 select * from tab2;
select now();

  5. 执行脚本，把其中的两个时间改成上一步中获取到的时间，然后就会发现内存占用情况：

    #!/bin/bash
#
for i in {1..20000}; do
  ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc1#root" --password "111" --pname "pitr11" --timestamp "2024-10-24 17:32:30" --account "acc1"
  mysql -h 127.0.0.1 -P 6001 -uacc1:root -p111 -e "select count(*) from db01.tab2;"
  sleep 1
  ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc1#root" --password "111" --pname "pitr11" --timestamp "2024-10-24 17:32:34" --account "acc1"
  mysql -h 127.0.0.1 -P 6001 -uacc1:root -p111 -e "select count(*) from db01.tab2;"
done
volgariver6 commented 1 day ago

@jensenojs please check this issue.

volgariver6 commented 1 day ago

the callstack is like which append Node_FUZZY_FILTER type node:

github.com/matrixorigin/matrixone/pkg/sql/plan.appendPrimaryConstraintPlan(0x140347d12d0, 0x1402bb72788, 0x1402be92000, 0x1402be84640, 0x0, {0x0, 0x0, 0x0}, {0x1403ad72480, 0x6, ...}, .      ..)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build_constraint_util.go:1464 +0x317c
github.com/matrixorigin/matrixone/pkg/sql/plan.makeOneInsertPlan({0x1051596f0?, 0x1402c3b4700?}, 0x140347d12d0, 0x1402bb72788, 0x1402be84640, 0x1402be92000, 0x0, 0x1, 0x1?, 0x0, ...)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build_dml_util.go:940 +0xf4
github.com/matrixorigin/matrixone/pkg/sql/plan.buildPreInsertRegularIndex(0x140389d01b0, {0x1051596f0, 0x1402c3b4700}, 0x140347d12d0, 0x1402bb72788, 0x1402bb6e000, 0x1402baa5800, 0x0, 0      x1403a50fad0, 0x140316ab6c0, ...)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build_dml_util.go:3886 +0x6c8
github.com/matrixorigin/matrixone/pkg/sql/plan.buildInsertPlansWithRelatedHiddenTable(0x140389d01b0, {0x1051596f0, 0x1402c3b4700}, 0x140347d12d0, 0x1402bb72788, 0x1402bb6e000, 0x1402baa      5800, 0x0, 0x0, 0x1, ...)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build_dml_util.go:846 +0x56c
github.com/matrixorigin/matrixone/pkg/sql/plan.buildInsertPlans({0x1051596f0, 0x1402c3b4700}, 0x140347d12d0, 0x1402bb72008, 0x140389d01b0, 0x1402bb6e000, 0x1402baa5800, 0x2?, 0x0, 0x140      3a50f110, ...)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build_dml_util.go:171 +0x2a0
github.com/matrixorigin/matrixone/pkg/sql/plan.buildInsert(0x140389d01b0, {0x1051596f0, 0x1402c3b4700}, 0x0, 0x0)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build_insert.go:313 +0x808
github.com/matrixorigin/matrixone/pkg/sql/plan.BuildPlan({0x1051596f0, 0x1402c3b4700}, {0x1050b6168, 0x140389d01b0}, 0x0)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/build.go:117 +0x1128
github.com/matrixorigin/matrixone/pkg/sql/plan.(*BaseOptimizer).Optimize(0x1401c9c5640, {0x1050b6168?, 0x140389d01b0?}, 0xa0?)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/sql/plan/optimize.go:56 +0x44
github.com/matrixorigin/matrixone/pkg/frontend.buildPlan({0x105092378, 0x1403a50e810}, {0x105180570, 0x14000d37b88}, {0x1051596f0, 0x1402c3b4700}, {0x1050b6168, 0x140389d01b0})
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/mysql_cmd_executor.go:1976 +0x5bc
github.com/matrixorigin/matrixone/pkg/frontend.(*TxnComputationWrapper).Compile(0x14038f37000, {0x10471f080?, 0x1402045c500}, 0x140364b18a8)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/computation_wrapper.go:194 +0x41c
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtInBack(0x14000d37b88, 0x1402045c500)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/back_exec.go:430 +0x3d4
github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt({0x105180570, 0x14000d37b88}, 0x1402045c500)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2635 +0x2ac
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt({0x105180570, 0x14000d37b88}, 0x1402045c500, {0x105159cf0, 0x1401c991508})
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2602 +0x370
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x105180570, 0x14000d37b88}, 0x1402045c500)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2550 +0x3e8
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x105180570, 0x14000d37b88}, 0x1402045c500)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2442 +0xe8
github.com/matrixorigin/matrixone/pkg/frontend.doComQueryInBack(0x14000d37b88, 0x1402045c500, 0x140347d11f0)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/back_exec.go:375 +0xc64
github.com/matrixorigin/matrixone/pkg/frontend.(*backExec).Exec(0x1402cf14020, {0x105092378, 0x1402f3ec2a0}, {0x1402c57f6c0, 0x66})
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/back_exec.go:133 +0x4f8
github.com/matrixorigin/matrixone/pkg/frontend.reCreateTableWithPitr({0x105092378, 0x1402f3ec2a0}, {0x1401775ef00, 0x24}, {0x1050eec78, 0x1402cf14020}, {0x1402c9aaa54, 0x6}, 0x180161594      97ea000, 0x14036585800)
    /Users/liubo/Workspace/Projects/MatrixOrigin/matrixone/pkg/frontend/pitr.go:1458 +0x604
volgariver6 commented 14 hours ago

pr committed