[Tech Request]: Umbrella issue of memory optimization

[sukki37]

Is there an existing issue for enhancement?

• [ ] I have checked the existing issues.

What would you like to be added?

We experienced many out-of-memory errors, leading to system instability. Memory management needs optimization for preventing OOM scenarios and for better control over memory usage. This issue is a summary of related bug and task issues:

• [x] #10509
• [x] #10945
• [x] #11225
• [x] #11221
• [x] #10659
• [x] #10212
• [ ] #7505
• [x] #9008
• [ ] #11509
• [ ] https://github.com/matrixorigin/MO-Cloud/issues/984
• [ ] https://github.com/matrixorigin/MO-Cloud/issues/952
• [ ] https://github.com/matrixorigin/MO-Cloud/issues/952
• [x] #10945
• [x] #12204
• [x] #12193
• [ ] https://github.com/matrixorigin/MO-Cloud/issues/1584
• [x] #11529

Dependencies:

• [x] #10370
• [x] #11511
• [x] #11650
• [x] #11651
• [x] #11652
• [ ] #3433
• [x] #11658
• [x] #11865
• [x] #11578
• [x] #10212

Why is this needed?

No response

Additional information

No response

[nnsgmsone]

处理daily run的问题中。。还未处理

[nnsgmsone]

讨论data race以及pipeline重构的一些事情，尚未处理

[nnsgmsone]

编写pipeline重构文档中，还未处理。

[nnsgmsone]

编写pipeline重构文档中，还未处理。

[nnsgmsone]

思考pipeline重构的设计中，还未处理

[nnsgmsone]

继续整理pipeline的文档中，还未处理

[nnsgmsone]

明天提交第一个pr，用于管理session的内存

[nnsgmsone]

session buffer的pr已经合入

[nnsgmsone]

先处理mpool的bug

[nnsgmsone]

先处理多session insert报错和daily run的问题

[nnsgmsone]

写软著中。。cache已经实现了一个，不过还没合入mo

[nnsgmsone]

在处理s-1。。。还未处理

[nnsgmsone]

在处理s-1。。。还未处理

[nnsgmsone]

开始编写代码，重新实现新的cache

[nnsgmsone]

明天开始评审第一个pr

[nnsgmsone]

继续修改代码中

[nnsgmsone]

今天在优先处理load的问题。

[nnsgmsone]

定位gap的内存(golang没有统计)中

[nnsgmsone]

继续处理中，具体的问题为https://github.com/matrixorigin/matrixone/issues/12731

[nnsgmsone]

等待测试，然后提交pr

[nnsgmsone]

继续处理mem分支的morpc相关的问题中。

[nnsgmsone]

经过测试eks和129都可以跑，https://github.com/matrixorigin/mo-auto-test/actions/runs/7015615926/job/19085534500。tke的stream closed在等复现，然后fix。

[nnsgmsone]

定位分支tpcc性能下降的问题中。

[nnsgmsone]

今天在修正#13219以及增加更多的metric中。

[nnsgmsone]

no process

[nnsgmsone]

no process

[nnsgmsone]

wait https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

wait https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

wait https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

正在和存储的同事协商https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

内存问题等待#12532

[nnsgmsone]

内存问题等待#12532

[nnsgmsone]

内存问题等待#12532

[nnsgmsone]

内存问题等待https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

内存问题等待https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

内存问题等待https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

内存问题等待https://github.com/matrixorigin/matrixone/issues/12532

[nnsgmsone]

no process

[nnsgmsone]

pr等待测试

[nnsgmsone]

pr等待测试和合入

[nnsgmsone]

等待pr合ru

[nnsgmsone]

等待pr和入

[nnsgmsone]

处理事务泄露中

[nnsgmsone]

定位问题中

[nnsgmsone]

no process

[nnsgmsone]

等待pr合入

[nnsgmsone]

cache pr已经合入，相关issue测试中

[nnsgmsone]

cache pr已经合入，相关issue测试中

[nnsgmsone]

这一系列以后远宁跟踪把

[ouyuanning]

待沟通跟进方式