[Bug]: mo killed by oom during stability test on standlone mode

[aressu1985]

Is there an existing issue for the same bug?

• [X] I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):69d60980cb6b81b86a33c8e7f8180b1fadd5bc12
- Hardware parameters:16C 64G
- OS type:
- Others:

Actual Behavior

during stability test on standlone, the mo was killed b oom : [Sat Jun 24 22:02:51 2023] [ 26370] 0 26370 265094 16433 450560 0 0 YDService [Sat Jun 24 22:02:51 2023] [ 58535] 0 58535 588310 4173 339968 0 0 sh [Sat Jun 24 22:02:51 2023] [3720175] 1000 3720175 3181 466 77824 0 0 run.sh [Sat Jun 24 22:02:51 2023] [2527897] 1000 2527897 3181 447 65536 0 0 run-helper.sh [Sat Jun 24 22:02:51 2023] [2527902] 1000 2527902 960475 17412 602112 0 0 Runner.Listener [Sat Jun 24 22:02:51 2023] [2631990] 1000 2631990 18543527 15107219 128045056 0 500 mo-service [Sat Jun 24 22:02:51 2023] [2634882] 1000 2634882 4280 470 73728 0 500 bash [Sat Jun 24 22:02:51 2023] [2635046] 1000 2635046 5561 1778 77824 0 500 bash [Sat Jun 24 22:02:51 2023] [2635189] 1000 2635189 4247 467 69632 0 500 bash [Sat Jun 24 22:02:51 2023] [2635199] 1000 2635199 5898738 76189 1683456 0 500 java [Sat Jun 24 22:02:51 2023] [2635421] 1000 2635421 4280 468 69632 0 500 bash [Sat Jun 24 22:02:51 2023] [2635427] 1000 2635427 10636863 406768 4804608 0 500 java [Sat Jun 24 22:02:51 2023] [2660982] 1000 2660982 1183993 110043 1384448 0 500 java [Sat Jun 24 22:02:51 2023] [2680104] 0 2680104 15460 2209 159744 0 0 barad_agent [Sat Jun 24 22:02:51 2023] [2680105] 0 2680105 313644 2919 311296 0 0 barad_agent [Sat Jun 24 22:02:51 2023] [2680205] 1000 2680205 5561 1572 73728 0 500 bash [Sat Jun 24 22:02:51 2023] [2680206] 1000 2680206 22528 317 143360 0 500 mysql [Sat Jun 24 22:02:51 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-14965.scope,task=mo-service,pid=2631990,uid=1000 [Sat Jun 24 22:02:51 2023] Out of memory: Killed process 2631990 (mo-service) total-vm:74174108kB, anon-rss:60428876kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:125044kB oom_score_adj:500

mo log: mo-service.log.tar.gz

the last 3 times proflies: 2023-06-24_21_38_20.tar.gz 2023-06-24_21_44_24.tar.gz 2023-06-24_21_50_26.tar.gz

Expected Behavior

No response

Steps to Reproduce

run a mo tester on stanlone and run the following test
1. bvt loop
   git clone mo-tester
   ./run.sh -n -g -p  xxxxx/matrixone/test/distributed/cases -e ddl -t 100

2. tpch loop 
   git clone mo-tpch, laod tpch 10G data
   ./run.sh -q all -s 10 -t 50000

3. sysbench mixed
   git clone mo-load, and init 10-talbes-100000-per-table 
   ./start.sh -m SYSBENCH -n 10 -s 100000
   ./start.sh -c cases/sysbench/mixed_10_100000/ -d 5000 -g

4. tpcc 10-10 long run
   git clone mo-tpcc, and load tpcc 10 warehouse data, and modify runMins to 5000 in props.mo
   ./runBenchmark.sh props.mo

Additional information

No response

[LeftHandCold]

[xzxiong]

watch

[qingxinhome]

watch

[daviszhen]

2023-06-24_21_38_20.tar.gz 里面的的mem profile。

[daviszhen]

2023-06-24_21_44_24.tar.gz 里面的mem profile。

[daviszhen]

2023-06-24_21_50_26.tar.gz 里面的mem profile。

[daviszhen]

这个大概率不是真实原因。goetty最近没改过。内存消耗应该跟以前一样大。

[fengttt]

@daviszhen your goetty switch merged. Now we are pulling from matrixorigin goetty repo. Please check if they are synced.

[fengttt]

9674

[daviszhen]

@daviszhen your goetty switch merged. Now we are pulling from matrixorigin goetty repo. Please check if they are synced.

They are same. I have checked it.

1. this issue's (69d60980cb6b81b86a33c8e7f8180b1fadd5bc12) goetty version is github.com/fagongzi/goetty/v2 v2.0.3-0.20230520035916-bc1fed6f5e26

2. the main : a67f9d959be976050d5b91d273bc6809ba822392

[daviszhen]

测试不能正常完成。1小时20分钟后，oom kill。

跑了半小时后，出了下面的问题：

1，tpch找不到表。

2，sysbench(mo-load)找不到表。

3，tpcc事务报错。

4，登录看。sysbench, tpch的库都不存在了

[daviszhen]

mo-tester的mo.yaml 中 builtindb: mo_catalog,tpcc_10,tpch_10g,sbtest,mo_task,information_schema,mysql,system_metrics,system,query_result

[cnutshell]

According to the OOM killer log, mo-service took 60428876kB memory, which was roughly 60GB memory. According to the memory profile of 2023-06-24_21_50_26.tar.gz, the inuse_space of mo-service is 22.78GB.

If mo-service didn't allocated memory repeatedly, it shouldn't be killed(22.78GB << 60GB). The reason why mo-service was killed by OOM should be: certain components of mo-service allocs memory so quickly that the scavenger of golang couldn't release free-memory back to OS in time.

Advice: just pay attention to those code that allocated too much memory:

[daviszhen]

昨天试了仅有tpcc,tpch,sysbench 混合跑的情况。

大概跑2小时后，内存oom。比带上bvt出oom的情况要慢。

最后几分中的内存情况。

[daviszhen]

期间watermark 的情况。

2023/06/26 19:38:26.576611 +0800 INFO logtail/mgr.go:160 init waterline to 1687779506576560208-0 2023/06/26 19:38:26.576674 +0800 INFO dn-service.logtail-push-service service/server.go:375 init waterline {"uuid": "dd4dccb4-4d3c-41f8-b482-5251dc7a41bf", "server-id": "6a0f9299-6294-4c42-be8f-8843a21c9174", "to": "PhysicalTime:1687779506576560208 "} 2023/06/26 19:39:28.837201 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 1073995872 2023/06/26 19:39:28.850698 +0800 INFO mpool/mpool.go:81 MPool pipeline-1b737384-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1073763840 2023/06/26 19:39:35.092109 +0800 INFO mpool/mpool.go:81 MPool pipeline-1eb9493c-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1073752760 2023/06/26 19:39:41.635527 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 2147520901 2023/06/26 19:39:45.355942 +0800 INFO mpool/mpool.go:81 MPool pipeline-2508d8d3-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1073799992 2023/06/26 19:40:06.946052 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 3221225512 2023/06/26 19:40:14.797836 +0800 INFO mpool/mpool.go:81 MPool pipeline-289aa2c4-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1073846024 2023/06/26 19:40:17.146592 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 4295107656 2023/06/26 19:41:42.267377 +0800 INFO mpool/mpool.go:81 MPool pipeline-6a62a06c-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1295272360 2023/06/26 19:42:00.353682 +0800 INFO mpool/mpool.go:81 MPool pipeline-760b0778-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1074287800 2023/06/26 19:42:03.983918 +0800 INFO mpool/mpool.go:81 MPool pipeline-77fa4d99-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1263125760 2023/06/26 19:42:07.163203 +0800 INFO mpool/mpool.go:81 MPool pipeline-77fa4d99-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 2206582016 2023/06/26 19:42:23.634502 +0800 INFO mpool/mpool.go:81 MPool pipeline-82fb492c-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1291712424 2023/06/26 19:42:52.508115 +0800 INFO mpool/mpool.go:81 MPool pipeline-94ba0441-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1076434104 2023/06/26 19:43:27.981915 +0800 INFO mpool/mpool.go:81 MPool pipeline-a7f3d62f-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1082656000 2023/06/26 19:43:40.874400 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 1075824672 2023/06/26 19:43:41.160988 +0800 INFO mpool/mpool.go:81 MPool pipeline-a7f3d62f-1416-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 2204910848 2023/06/26 19:43:56.866236 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 1073741856 2023/06/26 19:44:17.982667 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 5371429356 2023/06/26 19:44:48.475176 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 2156995744 2023/06/26 19:46:46.417191 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 3227241880 2023/06/26 19:47:08.559735 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 6447184088 2023/06/26 19:47:11.651694 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 2147566992 2023/06/26 19:48:02.132056 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 7518267680 2023/06/26 19:48:07.072031 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 4296428928 2023/06/26 19:49:01.434509 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 8595399833 2023/06/26 19:49:05.856064 +0800 INFO mpool/mpool.go:81 MPool pipeline-70183db6-1417-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1121776384 2023/06/26 19:49:06.264939 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 9668877078 2023/06/26 19:49:12.115521 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 3221394848 2023/06/26 19:50:19.511329 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 5371455448 2023/06/26 19:51:02.018465 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 10737420890 2023/06/26 19:51:06.923200 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 4295027384 2023/06/26 19:51:54.729821 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 11811190448 2023/06/26 19:52:22.150770 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 6442523408 2023/06/26 19:52:37.176088 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 12886583062 2023/06/26 19:52:51.874957 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 5368802136 2023/06/26 19:53:07.175586 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 13958689412 2023/06/26 19:53:46.959002 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 7518419104 2023/06/26 19:54:23.097671 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 15033106120 2023/06/26 19:54:26.654850 +0800 INFO mpool/mpool.go:81 MPool pipeline-2f27da9f-1418-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1085797120 2023/06/26 19:54:27.447032 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 16108075280 2023/06/26 19:55:01.701438 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 6442461328 2023/06/26 19:55:27.895723 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 8592318648 2023/06/26 19:55:58.832286 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 17181424378 2023/06/26 19:56:32.031103 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 18254357401 2023/06/26 19:56:34.620532 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 9664470944 2023/06/26 19:56:36.863270 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 7516203656 2023/06/26 19:57:01.784998 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 19336272680 2023/06/26 19:57:20.717683 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 10741074888 2023/06/26 19:57:38.829063 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 20401536576 2023/06/26 19:58:31.987669 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 21475736512 2023/06/26 19:58:42.299592 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 8589943568 2023/06/26 19:58:50.387510 +0800 INFO mpool/mpool.go:81 MPool pipeline-cd63f86b-1418-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1086386944 2023/06/26 19:58:50.500887 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 22570144544 2023/06/26 19:58:53.878373 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 11820817248 2023/06/26 19:59:50.615828 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 12888630792 2023/06/26 20:00:02.246466 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 23623682712 2023/06/26 20:00:31.384713 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 13964515248 2023/06/26 20:00:31.497299 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 24703743144 2023/06/26 20:00:52.145522 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 9663690624 2023/06/26 20:01:03.844787 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 25773894248 2023/06/26 20:01:22.585483 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 15036939936 2023/06/26 20:01:37.035832 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 27247965672 2023/06/26 20:02:35.469571 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 16110839632 2023/06/26 20:02:45.248514 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 27918348396 2023/06/26 20:02:54.140658 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 28996586174 2023/06/26 20:02:54.551932 +0800 INFO mpool/mpool.go:81 MPool pipeline-5f1c6272-1419-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1114114816 2023/06/26 20:03:20.077451 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 17181569928 2023/06/26 20:03:57.272672 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 10737447024 2023/06/26 20:04:01.810447 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 30064938677 2023/06/26 20:04:21.121478 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 18254143216 2023/06/26 20:04:52.369795 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 31157174736 2023/06/26 20:05:13.280508 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 19332839584 2023/06/26 20:05:30.188584 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 32322450656 2023/06/26 20:05:56.104789 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 20401967928 2023/06/26 20:06:05.541895 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 33286675460 2023/06/26 20:06:12.784490 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 34396009820 2023/06/26 20:06:12.952690 +0800 INFO mpool/mpool.go:81 MPool pipeline-d57ed4bc-1419-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1084584704 2023/06/26 20:06:46.037007 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 21478896472 2023/06/26 20:07:19.355030 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 35434083109 2023/06/26 20:07:21.759849 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 11811173152 2023/06/26 20:07:28.013861 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 22552185736 2023/06/26 20:07:53.951362 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 36514829760 2023/06/26 20:08:16.656783 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 37581082525 2023/06/26 20:08:28.454052 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 23626640536 2023/06/26 20:09:12.134095 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 24698503824 2023/06/26 20:09:22.430510 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 38660678896 2023/06/26 20:09:33.703943 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 39747661952 2023/06/26 20:09:33.894870 +0800 INFO mpool/mpool.go:81 MPool pipeline-4d55d394-141a-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1103131392 2023/06/26 20:10:06.450171 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 25776233504 2023/06/26 20:11:06.589064 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 40809705836 2023/06/26 20:11:12.281436 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 26846968376 2023/06/26 20:11:29.616769 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 41886128296 2023/06/26 20:11:31.730372 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 12884949048 2023/06/26 20:11:51.698090 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 27919249472 2023/06/26 20:12:15.256360 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 42968228704 2023/06/26 20:12:37.468716 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 44067237360 2023/06/26 20:12:37.516413 +0800 INFO mpool/mpool.go:81 MPool pipeline-bb3ae217-141a-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1075139328 2023/06/26 20:12:49.169376 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 28992850088 2023/06/26 20:13:29.314614 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 30069510096 2023/06/26 20:13:42.818146 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 45097275744 2023/06/26 20:14:10.646788 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 31145272536 2023/06/26 20:14:28.929796 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 46177787992 2023/06/26 20:15:23.401532 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 47288425080 2023/06/26 20:15:23.695629 +0800 INFO mpool/mpool.go:81 MPool pipeline-1e2ceb42-141b-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1101263616 2023/06/26 20:15:32.131829 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 32213135344 2023/06/26 20:16:18.799098 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 33288323144 2023/06/26 20:16:22.801335 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 48325548600 2023/06/26 20:17:08.652495 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 49685661104 2023/06/26 20:17:31.817586 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 34363010840 2023/06/26 20:18:07.549172 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 50490448000 2023/06/26 20:18:07.859352 +0800 INFO mpool/mpool.go:81 MPool pipeline-8043deb1-141b-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1101268992 2023/06/26 20:18:26.250435 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 35436589128 2023/06/26 20:19:37.059658 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 51542354464 2023/06/26 20:19:43.073237 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 36509294768 2023/06/26 20:20:11.496808 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 52618546992 2023/06/26 20:20:42.153495 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 37586514016 2023/06/26 20:21:15.110427 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 53688669175 2023/06/26 20:21:16.465278 +0800 INFO mpool/mpool.go:81 MPool pipeline-ef7dd040-141b-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1125659392 2023/06/26 20:21:18.536648 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 54761593432 2023/06/26 20:21:59.531257 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 38656000832 2023/06/26 20:22:52.776314 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 39730620664 2023/06/26 20:23:06.009165 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 56244611608 2023/06/26 20:23:51.796853 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 40804978952 2023/06/26 20:24:03.818573 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 56958284484 2023/06/26 20:24:03.980485 +0800 INFO mpool/mpool.go:81 MPool pipeline-54b78db6-141c-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1080619776 2023/06/26 20:24:50.494260 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 41878387200 2023/06/26 20:25:22.684597 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 57986593376 2023/06/26 20:25:40.003634 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 42956064152 2023/06/26 20:25:59.326564 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 59546250848 2023/06/26 20:26:50.311633 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 44024643448 2023/06/26 20:26:52.767368 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 60163132192 2023/06/26 20:26:53.020356 +0800 INFO mpool/mpool.go:81 MPool pipeline-b912f353-141c-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1107833600 2023/06/26 20:27:21.772418 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 13958668136 2023/06/26 20:27:36.139566 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 45098117576 2023/06/26 20:27:59.545078 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 61209354128 2023/06/26 20:28:41.561995 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 46172449384 2023/06/26 20:28:42.158790 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 62596131344 2023/06/26 20:29:34.540514 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 47248081976 2023/06/26 20:29:38.762300 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 63383416448 2023/06/26 20:29:38.966444 +0800 INFO mpool/mpool.go:81 MPool pipeline-1c35de5e-141d-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1085147904 2023/06/26 20:30:16.085642 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 48319002712 2023/06/26 20:30:33.547331 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 64483386288 2023/06/26 20:31:14.977538 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 49393967936 2023/06/26 20:31:20.529967 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 65498655463 2023/06/26 20:31:57.476875 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 50470342704 2023/06/26 20:32:03.884199 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 66572760576 2023/06/26 20:32:24.317212 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 67654853456 2023/06/26 20:32:24.646464 +0800 INFO mpool/mpool.go:81 MPool pipeline-7ee1789e-141d-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1077310208 2023/06/26 20:32:36.389535 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 51546152800 2023/06/26 20:33:27.455150 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 52617681704 2023/06/26 20:33:34.910393 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 68786140052 2023/06/26 20:34:02.316329 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 69801795928 2023/06/26 20:34:03.818850 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 53694003296 2023/06/26 20:34:40.581708 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 54761742096 2023/06/26 20:34:44.737406 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 70872917792 2023/06/26 20:35:00.899566 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 71942950720 2023/06/26 20:35:01.111446 +0800 INFO mpool/mpool.go:81 MPool pipeline-dcbe49fc-141d-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1098887936 2023/06/26 20:35:34.759365 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 55837503376 2023/06/26 20:36:25.397632 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 56909715472 2023/06/26 20:36:32.970111 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 73015725436 2023/06/26 20:36:46.619425 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 74309303376 2023/06/26 20:37:04.545276 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 57985601864 2023/06/26 20:37:41.252118 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 75173537312 2023/06/26 20:37:42.144960 +0800 INFO mpool/mpool.go:81 MPool pipeline-3bd45002-141e-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1090333440 2023/06/26 20:37:55.863548 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 59063184568 2023/06/26 20:38:37.563077 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 60132022376 2023/06/26 20:38:38.276369 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 76268910124 2023/06/26 20:39:10.266313 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 61203537400 2023/06/26 20:39:12.735583 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 77311323040 2023/06/26 20:39:30.120303 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 78593063224 2023/06/26 20:39:44.007936 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 62279699624 2023/06/26 20:40:23.214034 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 79482775792 2023/06/26 20:40:23.450495 +0800 INFO mpool/mpool.go:81 MPool pipeline-9cbea751-141e-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1120203520 2023/06/26 20:40:36.232137 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 63353335744 2023/06/26 20:41:14.059966 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 64432183624 2023/06/26 20:41:15.960615 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 80530986886 2023/06/26 20:41:47.510785 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 81675308700 2023/06/26 20:41:48.322042 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 65500449616 2023/06/26 20:42:13.769561 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 82683941212 2023/06/26 20:42:35.560540 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 66577631232 2023/06/26 20:43:04.869695 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 83795225104 2023/06/26 20:43:05.166030 +0800 INFO mpool/mpool.go:81 MPool pipeline-fc9f4186-141e-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1086206720 2023/06/26 20:43:27.788458 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 67649503360 2023/06/26 20:44:03.090647 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 84831354004 2023/06/26 20:44:08.357743 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 68722887296 2023/06/26 20:44:43.449646 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 85902014508 2023/06/26 20:44:59.687997 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 69795539984 2023/06/26 20:45:32.559152 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 86973718812 2023/06/26 20:45:41.010016 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 70871317088 2023/06/26 20:45:45.650469 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 88050768708 2023/06/26 20:45:45.672257 +0800 INFO mpool/mpool.go:81 MPool pipeline-5cc36c57-141f-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1115859712 2023/06/26 20:46:27.799548 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 71943043752 2023/06/26 20:46:51.784275 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 89124072532 2023/06/26 20:46:59.875678 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 73018882744 2023/06/26 20:47:33.875612 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 90194848522 2023/06/26 20:47:56.997007 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 74091470840 2023/06/26 20:48:25.156402 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 91281035052 2023/06/26 20:48:29.564199 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 92353153673 2023/06/26 20:48:29.931283 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 75162000000 2023/06/26 20:48:29.943228 +0800 INFO mpool/mpool.go:81 MPool pipeline-be531620-141f-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1105163008 2023/06/26 20:49:06.476055 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 76236096856 2023/06/26 20:49:43.248441 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 93420248764 2023/06/26 20:49:52.889248 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 77313612480 2023/06/26 20:50:12.922985 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 94964064873 2023/06/26 20:50:35.481580 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 78389842488 2023/06/26 20:51:06.293509 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 95577094380 2023/06/26 20:51:08.969684 +0800 INFO mpool/mpool.go:81 MPool pipeline-1d38f64f-1420-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1116058368 2023/06/26 20:51:09.045404 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 96675117260 2023/06/26 20:51:20.454791 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 79458062304 2023/06/26 20:52:01.205036 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 80530733320 2023/06/26 20:52:41.489619 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 97712349004 2023/06/26 20:52:54.009545 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 81607046808 2023/06/26 20:53:17.322452 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 98786286028 2023/06/26 20:53:35.765681 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 82679830152 2023/06/26 20:53:46.441177 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 99918188092 2023/06/26 20:53:46.686986 +0800 INFO mpool/mpool.go:81 MPool pipeline-7b250a04-1420-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1117506304 2023/06/26 20:54:15.288978 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 83755541272 2023/06/26 20:54:39.791382 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 100935372364 2023/06/26 20:54:51.117999 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 84825822224 2023/06/26 20:55:12.956077 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 102008125120 2023/06/26 20:55:36.069675 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 103099226563 2023/06/26 20:55:39.335524 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 85899692504 2023/06/26 20:56:25.751610 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 104183512284 2023/06/26 20:56:25.798491 +0800 INFO mpool/mpool.go:81 MPool pipeline-da47663c-1420-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1119734528 2023/06/26 20:56:35.496525 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 86974522536 2023/06/26 20:57:12.692192 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 88053372856 2023/06/26 20:57:39.806851 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 105228753668 2023/06/26 20:57:53.675660 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 89125286552 2023/06/26 20:58:10.094432 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 106816103164 2023/06/26 20:58:46.319607 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 90195125144 2023/06/26 20:59:04.514619 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 107382068172 2023/06/26 20:59:06.729171 +0800 INFO mpool/mpool.go:81 MPool pipeline-3a20123c-1421-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1118204672 2023/06/26 20:59:07.361724 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 108448674011 2023/06/26 20:59:30.656660 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 91270571208 2023/06/26 21:00:11.543965 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 92347518280 2023/06/26 21:00:47.487559 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 109522384684 2023/06/26 21:01:09.967595 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 93416100752 2023/06/26 21:01:30.390266 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 110648149684 2023/06/26 21:01:49.889811 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 94491121792 2023/06/26 21:01:52.644591 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 111674185580 2023/06/26 21:01:52.849226 +0800 INFO mpool/mpool.go:81 MPool pipeline-9ce92e1a-1421-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1085141760 2023/06/26 21:02:27.398999 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 95564660400 2023/06/26 21:02:47.358672 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 112802586700 2023/06/26 21:03:14.118899 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 96636864304 2023/06/26 21:03:32.510025 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 113817694880 2023/06/26 21:03:57.718613 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 97711582000 2023/06/26 21:04:07.774055 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 114898000112 2023/06/26 21:04:35.971875 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 98789664816 2023/06/26 21:04:36.308081 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 116019532908 2023/06/26 21:04:36.620436 +0800 INFO mpool/mpool.go:81 MPool pipeline-fe7a754e-1421-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1118548736 2023/06/26 21:05:29.192551 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 99863760504 2023/06/26 21:05:44.800426 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 117042717228 2023/06/26 21:06:08.853563 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 100935639432 2023/06/26 21:06:16.220425 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 118120926300 2023/06/26 21:06:43.394279 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 102014940976 2023/06/26 21:06:56.336318 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 119185924332 2023/06/26 21:07:15.049875 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 120277743980 2023/06/26 21:07:15.240357 +0800 INFO mpool/mpool.go:81 MPool pipeline-5cb71b5d-1422-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1118554880 2023/06/26 21:07:22.968513 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 103084179504 2023/06/26 21:08:05.602745 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 104154343888 2023/06/26 21:08:26.928685 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 121332920132 2023/06/26 21:08:47.891634 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 105227743128 2023/06/26 21:08:54.379955 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 122470850744 2023/06/26 21:09:26.139375 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 106300543416 2023/06/26 21:09:35.093057 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 123482378076 2023/06/26 21:09:53.304395 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 124597747756 2023/06/26 21:09:53.600580 +0800 INFO mpool/mpool.go:81 MPool pipeline-bb703813-1422-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1085229824 2023/06/26 21:10:12.475923 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 107375492032 2023/06/26 21:10:59.459568 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 108449445552 2023/06/26 21:11:06.646451 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 125706935768 2023/06/26 21:11:36.379740 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 127064031684 2023/06/26 21:11:43.085734 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 109525482264 2023/06/26 21:12:27.266020 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 110597473224 2023/06/26 21:12:33.353251 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 127779939860 2023/06/26 21:12:36.355706 +0800 INFO mpool/mpool.go:81 MPool pipeline-1c5cac19-1423-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1077236480 2023/06/26 21:12:36.419191 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 128864788252 2023/06/26 21:13:09.440011 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 111675683136 2023/06/26 21:13:56.115552 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 112743576480 2023/06/26 21:13:56.770357 +0800 INFO mpool/mpool.go:81 MPool taeVectorPool new high watermark high water mark : 15032827800 2023/06/26 21:14:01.786514 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 129923722882 2023/06/26 21:14:30.116543 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 131365162444 2023/06/26 21:14:34.337963 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 113820753176 2023/06/26 21:15:30.436119 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 132130940188 2023/06/26 21:15:30.635887 +0800 INFO mpool/mpool.go:81 MPool pipeline-8513a54d-1423-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1078315776 2023/06/26 21:15:43.956076 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 114892414784 2023/06/26 21:16:26.666993 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 115966516944 2023/06/26 21:16:33.399004 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 133145557876 2023/06/26 21:17:01.897961 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 117041079664 2023/06/26 21:17:10.023848 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 134227994668 2023/06/26 21:17:59.870522 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 118113683800 2023/06/26 21:18:07.019899 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 135295378172 2023/06/26 21:18:08.959181 +0800 INFO mpool/mpool.go:81 MPool pipeline-e2ed26d0-1423-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1074795264 2023/06/26 21:18:08.964217 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 136369212516 2023/06/26 21:19:00.555493 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 119185780320 2023/06/26 21:19:40.834262 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 120261063800 2023/06/26 21:19:45.276899 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 137441219228 2023/06/26 21:20:28.490634 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 121333400648 2023/06/26 21:20:30.819496 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 138577752316 2023/06/26 21:20:51.506824 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 139612742052 2023/06/26 21:20:51.610626 +0800 INFO mpool/mpool.go:81 MPool pipeline-43e5b54e-1424-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1139987200 2023/06/26 21:21:07.174849 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 122410275720 2023/06/26 21:21:41.508054 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 123484118544 2023/06/26 21:21:45.247397 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 140662554176 2023/06/26 21:22:19.785120 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 141734684588 2023/06/26 21:22:19.785197 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 124556912776 2023/06/26 21:23:22.467440 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 142808085916 2023/06/26 21:23:24.296788 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 125632526064 2023/06/26 21:23:36.533559 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 143888450500 2023/06/26 21:23:36.562793 +0800 INFO mpool/mpool.go:81 MPool pipeline-a5b4121c-1424-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1089794816 2023/06/26 21:24:10.146574 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 126706509480 2023/06/26 21:24:49.121562 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 127776032392 2023/06/26 21:24:53.046703 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 144955551373 2023/06/26 21:25:22.068727 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 146323614660 2023/06/26 21:25:42.731408 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 128851186864 2023/06/26 21:26:13.573587 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 147168623106 2023/06/26 21:26:13.838646 +0800 INFO mpool/mpool.go:81 MPool pipeline-038e821a-1425-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1099844352 2023/06/26 21:26:16.559956 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 148182512100 2023/06/26 21:26:23.581717 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 129927695240 2023/06/26 21:27:08.641023 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 130998173544 2023/06/26 21:27:57.435766 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 149250637564 2023/06/26 21:28:06.130989 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 132071808288 2023/06/26 21:28:49.877492 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 133147543632 2023/06/26 21:28:53.070421 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 150325434916 2023/06/26 21:28:56.785919 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 151422290495 2023/06/26 21:28:56.961012 +0800 INFO mpool/mpool.go:81 MPool pipeline-64870244-1425-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1120875264 2023/06/26 21:29:32.251421 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 134221201760 2023/06/26 21:30:17.363843 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 135297049736 2023/06/26 21:30:18.759142 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 152471365929 2023/06/26 21:30:42.857487 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 153699512284 2023/06/26 21:31:07.411994 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 136367423440 2023/06/26 21:31:38.016335 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 154625766388 2023/06/26 21:31:40.213519 +0800 INFO mpool/mpool.go:81 MPool pipeline-c63338f7-1425-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1119636224 2023/06/26 21:31:40.458198 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 155734312148 2023/06/26 21:31:42.330779 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 137449708512 2023/06/26 21:32:30.508948 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 138515487056 2023/06/26 21:33:14.165506 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 156769065044 2023/06/26 21:33:18.881007 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 139587378800 2023/06/26 21:33:58.165212 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 157848470276 2023/06/26 21:34:03.780823 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 140663760712 2023/06/26 21:34:22.118481 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 158961923388 2023/06/26 21:34:22.501538 +0800 INFO mpool/mpool.go:81 MPool pipeline-26e7460f-1426-11ee-bf5c-7070fc0034e7 new high watermark high water mark : 1122104064 2023/06/26 21:34:42.929065 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 141735051080 2023/06/26 21:35:20.860254 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 159991181068 2023/06/26 21:35:27.703471 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 142812096400 2023/06/26 21:36:05.879998 +0800 INFO mpool/mpool.go:81 MPool global new high watermark high water mark : 161061301782 2023/06/26 21:36:12.952202 +0800 INFO mpool/mpool.go:81 MPool distributed_tae new high watermark high water mark : 143889170192

[daviszhen]

中午用 dacfe86b80aba01df84e3c25ff3dce55aa27f26c. tpcc+tpch+sysbench。

跑了2个小时，没出现oom。内存控制30G左右。

[daviszhen]

试验了 23日最后一个版本。 f0e71a0fe33756d2242f33d41b32356a249412be 跑了1个小时，内存有泄漏的现象出现。

[daviszhen]

试验了22日最后一个版本。 bf23f2a999702b465669249f95a9c4bf8bf37322 oom了。超过2个半小时。

[daviszhen]

昨晚试验了版本 c5a8d237480b963043a3872d94de4256a953bec4 没跑出来oom。（从 2023/06/27 20:31:12 到 2023/06/28 09:50:15）。

[daviszhen]

中午跑了，26日最后一个版本。 9084cc44dfda39cc223ad408c267434f467b1585 没跑出来oom。超过3个小时。（2023/06/28 10:28:52 ~ 2023/06/28 14:05:08) 之前2个半小时基本能跑出oom了。

[daviszhen]

下午跑25日，最后一个版本。 ca8f93c3485bbb813d0c8f6c492b67b3abcbbf9a 没跑出来oom。超过4个小时。（2023/06/28 14:18:26 ~ 2023/06/28 19:02:04)

[daviszhen]

昨晚跑了。4f3c9a462 跑出mpool oom，然后panic了。（2023/06/28 19:07:34 ～ 2023/06/29 02:00:08）

engzhen@pengzhen:~/Documents/matrixone$ ./mo-service -debug-http 127.0.0.1:8123 -launch etc/launch-tae-CN-tae-DN/launch.toml > log.txt 
[mysql] 2023/06/28 19:33:20 packets.go:37: read tcp 127.0.0.1:58604->127.0.0.1:6001: i/o timeout
[mysql] 2023/06/29 01:50:54 packets.go:37: read tcp 127.0.0.1:60236->127.0.0.1:6001: i/o timeout
panic: unexpected const expr isnull:true 

goroutine 196086735 [running]:
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.getCompositeFilterFuncByExpr(0x1?, 0xa0?)
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/util.go:1020 +0x60e
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getCompositPKFilter(0xc1b33adcc0, 0x1dd69af?)
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:159 +0x1d7
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getReadFilter(0xc7ba9bad30?, 0xe73400fc7070318a?)
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:132 +0x3b
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*blockMergeReader).Read(0xc1b33adcc0, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0xc04e2f58a8?, 0x17a38ff?, {0x2ec5690, 0xc29727c500})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:437 +0x768
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc2ad04a0c0, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0xc2b89f3401?, 0x55e9ea?, {0x2ec5690, 0xc29727c500})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc2ad04a0a8, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0xc1b32436f8?, 0x0?, {0x2ec5690, 0xc29727c500})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc2ad04a0d8, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0x10000000013?, 0x7f7ca16f5cd8?, {0x2ec5690, 0xc29727c500})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/pipeline.(*Pipeline).Run(0xc36c0cbb70, {0x2ec53c0, 0xc2ad04a0d8}, 0xc29727c500)
    /home/pengzhen/Documents/matrixone/pkg/vm/pipeline/pipeline.go:79 +0x2bd
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).Run(0xc05e9c3a00, 0xc12f7c9550?)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:84 +0x16e
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).ParallelRun(0xc05e9c3a00, 0xc283870480, 0x0)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:340 +0xba5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).RemoteRun(0xc05e9c3a00, 0xc283870480)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:188 +0xa5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun.func3(0x0?)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:122 +0x2f
created by github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:121 +0x1f8
panic: unexpected const expr isnull:true 

goroutine 196085659 [running]:
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.getCompositeFilterFuncByExpr(0x1?, 0xa0?)
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/util.go:1020 +0x60e
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getCompositPKFilter(0xc185823a40, 0x1dd69af?)
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:159 +0x1d7
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getReadFilter(0xc00e1bed30?, 0xe73400fc7070318a?)
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:132 +0x3b
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*blockMergeReader).Read(0xc185823a40, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0xc14c9d18a8?, 0x17a38ff?, {0x2ec5690, 0xc1e3442a00})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:437 +0x768
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc0b130d9c8, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0x4231ab?, 0x55e9ea?, {0x2ec5690, 0xc1e3442a00})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc0b130d9b0, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0xc18566e0a0?, 0x0?, {0x2ec5690, 0xc1e3442a00})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc0b130d9f8, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0x13?, 0x7f7cf40cf6e8?, {0x2ec5690, 0xc1e3442a00})
    /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/pipeline.(*Pipeline).Run(0xc005885b70, {0x2ec53c0, 0xc0b130d9f8}, 0xc1e3442a00)
    /home/pengzhen/Documents/matrixone/pkg/vm/pipeline/pipeline.go:79 +0x2bd
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).Run(0xc8324729c0, 0xc5ac074030?)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:84 +0x16e
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).ParallelRun(0xc8324729c0, 0xc0c91ff200, 0x0)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:340 +0xba5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).RemoteRun(0xc8324729c0, 0xc0c91ff200)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:188 +0xa5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun.func3(0x0?)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:122 +0x2f
created by github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:121 +0x1f8

mo日志

2023/06/29 02:00:08.883740 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.494179069s connectionId 164|127.0.0.1:39282|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 550244|95eebd0e-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.887691 +0800 INFO jobs/compactblk.go:341 [Done] {"txn-start-ts": "1687975208614982927-0", "operation": "[671525]compact", "compacted": "[NA]BLK[<272468-272475-9bc61921-15d9-11ee-8a31-7070fc0034e7-737-0>]", "created": "BLK<272468-272475-9bc61921-15d9-11ee-8a31-7070fc0034e7-871-0>", "compactedRows": 1339, "TotalChanges": 4, "createdRows": 1335, "duration": "272.513019ms"}
2023/06/29 02:00:08.891554 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.168892834s connectionId 162|127.0.0.1:39264|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 540573|9576fd5a-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.953579 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.263581861s connectionId 137|127.0.0.1:51902|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 455180|8eacafe5-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.958348 +0800 ERROR txnimpl/antidepend.go:42 error: r-w conflict
2023/06/29 02:00:08.958460 +0800 INFO txnimpl/table.go:289 [Start] {"txn-start-ts": "1687975184145508524-1", "operation": "transfer-deletes", "operand": "BLK<272468-272487-e6a01ab9-15dc-11ee-8a31-7070fc0034e7-0-0>", "phase": "Phase_Freeze"}
2023/06/29 02:00:08.958527 +0800 INFO txnimpl/table.go:297 [End] {"txn-start-ts": "1687975184145508524-1", "operation": "transfer-deletes", "operand": "BLK<272468-272487-e6a01ab9-15dc-11ee-8a31-7070fc0034e7-0-0>", "phase": "Phase_Freeze"}
2023/06/29 02:00:08.965547 +0800 INFO rpc/handle.go:147 Commit with long latency {"duration": "166.823139ms", "debug": "db2d737158184e7ba06ab065356364ba/Active/S:1687975184145508524-1"}
2023/06/29 02:00:08.968689 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.195791826s connectionId 145|127.0.0.1:39116|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 483989|91e6bc78-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.969146 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
     allocations : 5348297287
     frees : 2712790184
     alloc bytes : 13412320449408
     free bytes : 12312788353547
     current bytes : 1099532095861
     high water mark : 1099532095885

2023/06/29 02:00:08.969196 +0800 ERROR mpool/mpool.go:539 error: error: out of memory
2023/06/29 02:00:08.969214 +0800 ERROR mpool/mpool.go:539 error: error: out of memory
2023/06/29 02:00:08.970351 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970410 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970519 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970544 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970580 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970623 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970656 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970687 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970691 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970716 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970726 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970756 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970779 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970806 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970811 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.969202 +0800 ERROR mpool/mpool.go:539 error: error: out of memory
2023/06/29 02:00:08.969196 +0800 ERROR mpool/mpool.go:539 error: error: out of memory

[daviszhen]

上午跑了。 875669ff4eee526eed0baa607cd4545341405c99 超过4小时。没有oom。(2023/06/29 09:48:55 ~ 2023/06/29 14:09:18)

[daviszhen]

下午跑。 e2d0da38b3b64dee6915caebdd1593661378fdae 超过4小时。没有oom（2023/06/29 15:02:19 ~ 2023/06/29 19:35:40）

[daviszhen]

昨晚上跑 6c14bc30b09ef54c53f0fb7e992fb1343ecd0c76 oom。（2023/06/29 19:43:09 ～ 2023/06/29 21:54:20）

[daviszhen]

经过上面二分测试。

1，实际验证了25日、26日、27日的最后一个版本是没有oom的。

2, 明确了oom改掉的版本。

[daviszhen]

昨晚跑了。4f3c9a462 跑出mpool oom，然后panic了。（2023/06/28 19:07:34 ～ 2023/06/29 02:00:08）

engzhen@pengzhen:~/Documents/matrixone$ ./mo-service -debug-http 127.0.0.1:8123 -launch etc/launch-tae-CN-tae-DN/launch.toml > log.txt 
[mysql] 2023/06/28 19:33:20 packets.go:37: read tcp 127.0.0.1:58604->127.0.0.1:6001: i/o timeout
[mysql] 2023/06/29 01:50:54 packets.go:37: read tcp 127.0.0.1:60236->127.0.0.1:6001: i/o timeout
panic: unexpected const expr isnull:true 

goroutine 196086735 [running]:
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.getCompositeFilterFuncByExpr(0x1?, 0xa0?)
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/util.go:1020 +0x60e
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getCompositPKFilter(0xc1b33adcc0, 0x1dd69af?)
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:159 +0x1d7
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getReadFilter(0xc7ba9bad30?, 0xe73400fc7070318a?)
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:132 +0x3b
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*blockMergeReader).Read(0xc1b33adcc0, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0xc04e2f58a8?, 0x17a38ff?, {0x2ec5690, 0xc29727c500})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:437 +0x768
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc2ad04a0c0, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0xc2b89f3401?, 0x55e9ea?, {0x2ec5690, 0xc29727c500})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc2ad04a0a8, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0xc1b32436f8?, 0x0?, {0x2ec5690, 0xc29727c500})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc2ad04a0d8, {0x2ed7118, 0xc46c0b4060}, {0xc27cff1880, 0xe, 0xe}, 0x10000000013?, 0x7f7ca16f5cd8?, {0x2ec5690, 0xc29727c500})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/pipeline.(*Pipeline).Run(0xc36c0cbb70, {0x2ec53c0, 0xc2ad04a0d8}, 0xc29727c500)
  /home/pengzhen/Documents/matrixone/pkg/vm/pipeline/pipeline.go:79 +0x2bd
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).Run(0xc05e9c3a00, 0xc12f7c9550?)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:84 +0x16e
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).ParallelRun(0xc05e9c3a00, 0xc283870480, 0x0)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:340 +0xba5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).RemoteRun(0xc05e9c3a00, 0xc283870480)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:188 +0xa5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun.func3(0x0?)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:122 +0x2f
created by github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:121 +0x1f8
panic: unexpected const expr isnull:true 

goroutine 196085659 [running]:
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.getCompositeFilterFuncByExpr(0x1?, 0xa0?)
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/util.go:1020 +0x60e
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getCompositPKFilter(0xc185823a40, 0x1dd69af?)
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:159 +0x1d7
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*withFilterMixin).getReadFilter(0xc00e1bed30?, 0xe73400fc7070318a?)
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:132 +0x3b
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*blockMergeReader).Read(0xc185823a40, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0xc14c9d18a8?, 0x17a38ff?, {0x2ec5690, 0xc1e3442a00})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:437 +0x768
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc0b130d9c8, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0x4231ab?, 0x55e9ea?, {0x2ec5690, 0xc1e3442a00})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc0b130d9b0, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0xc18566e0a0?, 0x0?, {0x2ec5690, 0xc1e3442a00})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*mergeReader).Read(0xc0b130d9f8, {0x2ed7118, 0xc04c817350}, {0xc16d8ffcc0, 0x13, 0x13}, 0x13?, 0x7f7cf40cf6e8?, {0x2ec5690, 0xc1e3442a00})
  /home/pengzhen/Documents/matrixone/pkg/vm/engine/disttae/reader.go:479 +0x10c
github.com/matrixorigin/matrixone/pkg/vm/pipeline.(*Pipeline).Run(0xc005885b70, {0x2ec53c0, 0xc0b130d9f8}, 0xc1e3442a00)
  /home/pengzhen/Documents/matrixone/pkg/vm/pipeline/pipeline.go:79 +0x2bd
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).Run(0xc8324729c0, 0xc5ac074030?)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:84 +0x16e
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).ParallelRun(0xc8324729c0, 0xc0c91ff200, 0x0)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:340 +0xba5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).RemoteRun(0xc8324729c0, 0xc0c91ff200)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:188 +0xa5
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun.func3(0x0?)
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:122 +0x2f
created by github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun
  /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:121 +0x1f8

mo日志

2023/06/29 02:00:08.883740 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.494179069s connectionId 164|127.0.0.1:39282|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 550244|95eebd0e-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.887691 +0800 INFO jobs/compactblk.go:341 [Done] {"txn-start-ts": "1687975208614982927-0", "operation": "[671525]compact", "compacted": "[NA]BLK[<272468-272475-9bc61921-15d9-11ee-8a31-7070fc0034e7-737-0>]", "created": "BLK<272468-272475-9bc61921-15d9-11ee-8a31-7070fc0034e7-871-0>", "compactedRows": 1339, "TotalChanges": 4, "createdRows": 1335, "duration": "272.513019ms"}
2023/06/29 02:00:08.891554 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.168892834s connectionId 162|127.0.0.1:39264|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 540573|9576fd5a-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.953579 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.263581861s connectionId 137|127.0.0.1:51902|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 455180|8eacafe5-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.958348 +0800 ERROR txnimpl/antidepend.go:42 error: r-w conflict
2023/06/29 02:00:08.958460 +0800 INFO txnimpl/table.go:289 [Start] {"txn-start-ts": "1687975184145508524-1", "operation": "transfer-deletes", "operand": "BLK<272468-272487-e6a01ab9-15dc-11ee-8a31-7070fc0034e7-0-0>", "phase": "Phase_Freeze"}
2023/06/29 02:00:08.958527 +0800 INFO txnimpl/table.go:297 [End] {"txn-start-ts": "1687975184145508524-1", "operation": "transfer-deletes", "operand": "BLK<272468-272487-e6a01ab9-15dc-11ee-8a31-7070fc0034e7-0-0>", "phase": "Phase_Freeze"}
2023/06/29 02:00:08.965547 +0800 INFO rpc/handle.go:147 Commit with long latency {"duration": "166.823139ms", "debug": "db2d737158184e7ba06ab065356364ba/Active/S:1687975184145508524-1"}
2023/06/29 02:00:08.968689 +0800 INFO frontend/util.go:530 time of Exec.Run : 1.195791826s connectionId 145|127.0.0.1:39116|{account sys:dump:moadmin -- 0:1:0}|goRoutineId 483989|91e6bc78-15a4-11ee-8a31-7070fc0034e7
2023/06/29 02:00:08.969146 +0800 INFO mpool/mpool.go:81 MPool global new high watermark
     allocations : 5348297287
     frees : 2712790184
     alloc bytes : 13412320449408
     free bytes : 12312788353547
     current bytes : 1099532095861
     high water mark : 1099532095885

2023/06/29 02:00:08.969196 +0800 ERROR mpool/mpool.go:539 error: error: out of memory
2023/06/29 02:00:08.969214 +0800 ERROR mpool/mpool.go:539 error: error: out of memory
2023/06/29 02:00:08.970351 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970410 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970519 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970544 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970580 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970623 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970656 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970687 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970691 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970716 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970726 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970756 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970779 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970806 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.970811 +0800 WARN connector/connector.go:50 reg.Ctx done during connector send
2023/06/29 02:00:08.969202 +0800 ERROR mpool/mpool.go:539 error: error: out of memory
2023/06/29 02:00:08.969196 +0800 ERROR mpool/mpool.go:539 error: error: out of memory

目前还遗留这个问题。

[daviszhen]

与 @aressu1985 沟通后，先走给他。

[aressu1985]

still exists for commit : d19e2b0a2b35d217c64d72e7bd1d07a00890b55a

[fengttt]

Please do not reassign bug to the cat.

[daviszhen]

继续profile

[daviszhen]

still exists for commit : d19e2b0

这个版本我启动不了。

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x202bdfc]

goroutine 1275 [running]:
github.com/matrixorigin/matrixone/pkg/cnservice/cnclient.IsCNClientReady()
    /home/pengzhen/Documents/matrixone/pkg/cnservice/cnclient/client.go:50 +0x3c
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).RemoteRun(0xc004f33c70, 0xc003aead80)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:186 +0x49
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun.func3(0xc005e0a7b0?)
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:122 +0x2f
created by github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun
    /home/pengzhen/Documents/matrixone/pkg/sql/compile/scope.go:121 +0x1f8

[daviszhen]

在跑测试，还未得出有效结论

[daviszhen]

没抓到oom前有效的profile。今晚再试试

[daviszhen]

今天没出现oom。 从11:03分开始到21:42。

[daviszhen]

从07/10 11:03:47 到 07/11 21:24:50 没有oom。但是内存确实泄漏了。从20g 到 50g

[daviszhen]

没有oom。 从2023/07/10 11:03:47 到 2023/07/12 21:01:14

没有oom。内存在48g。

[aressu1985]

repro on 7.13 the profiles for last 1 minutes is : profile.tar.gz

[daviszhen]

repro on 7.13 the profiles for last 1 minutes is : profile.tar.gz

从6个profile中heap的top情况看出。

objectio.constructorFactory.func1.1 在oom前爆涨。从3G涨到8G。

[daviszhen]

等存储的修改。

[LeftHandCold]

这个case cn的cache增大配置2G就可以了。原因是订阅多个table时会有多个线程同时loadcheckpoint，但是现在load checkpoint没有过滤，导致每一个协程都会把4 500M的checkpoint数据都读上来，然后根据tableid过滤返回数据。默认cn的cache 512M，加上prefetch，所以cn的cache永远不能覆盖，所以内存暴涨，当前的workaround，调整cn的cahce为2G即可，后续会有一个pr来优化这一点。

[qingxinhome]

in progress

[daviszhen]

等存储的修改。

[daviszhen]

等存储的修改。

[sukki37]

depend on #10681

[LeftHandCold]

还在开发

[LeftHandCold]

已过BVT，还需要做稳定性测试

[LeftHandCold]

还在测试性能和稳定性

[LeftHandCold]

在修复性能问题

[LeftHandCold]

自测中