matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 276 forks source link

[Tech Request]: too many allocations in scan #18873

Closed badboynt1 closed 1 month ago

badboynt1 commented 1 month ago

Is there an existing issue for the same tech request?

Does this tech request not affect user experience?

What would you like to be added ?

跑一轮tpch1T, fileservice相关的堆外分配大约有5.5T。 明显不应该这么多,因为现在scan算子可以复用自己的内存。

(pprof) top 50 --cum
Showing nodes accounting for 6142.95GB, 99.75% of 6158.53GB total
Dropped 159 nodes (cum <= 30.79GB)
      flat  flat%   sum%        cum   cum%
 5542.88GB 90.00% 90.00%  5542.88GB 90.00%  | stack omitted |
         0     0% 90.00%   615.65GB 10.00%  runtime.goexit
         0     0% 90.00%   615.41GB  9.99%  github.com/panjf2000/ants/v2.(*goWorker).run.func1
         0     0% 90.00%   615.37GB  9.99%  github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).MergeRun.func1
         0     0% 90.00%      615GB  9.99%  github.com/matrixorigin/matrixone/pkg/sql/compile.(*Scope).Run
         0     0% 90.00%      615GB  9.99%  github.com/matrixorigin/matrixone/pkg/vm/pipeline.(*Pipeline).run
         0     0% 90.00%   605.09GB  9.83%  github.com/matrixorigin/matrixone/pkg/vm.ChildrenCall
         0     0% 90.00%   605.05GB  9.82%  github.com/matrixorigin/matrixone/pkg/sql/colexec/projection.(*Projection).Call
  600.08GB  9.74% 99.75%   600.08GB  9.74%  github.com/matrixorigin/matrixone/pkg/common/malloc.(*MetricsAllocator[...]).Allocate
         0     0% 99.75%   600.08GB  9.74%  github.com/matrixorigin/matrixone/pkg/common/malloc.(*RandomAllocator[...]).Allocate
         0     0% 99.75%   600.08GB  9.74%  github.com/matrixorigin/matrixone/pkg/fileservice.(*LocalFS).Read
         0     0% 99.75%   600.08GB  9.74%  github.com/matrixorigin/matrixone/pkg/fileservice.(*LocalFS).read
         0     0% 99.75%   599.47GB  9.73%  github.com/matrixorigin/matrixone/pkg/objectio.ReadOneBlockWithMeta
         0     0% 99.75%   599.47GB  9.73%  github.com/matrixorigin/matrixone/pkg/objectio.ReadOneBlock
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/pipeline.(*Pipeline).Run
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/sql/colexec/table_scan.(*TableScan).Call
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*reader).Read
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.BlockDataRead
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.LoadColumnsData
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.LoadColumns
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.readBlockData
         0     0% 99.75%   599.43GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.readBlockData.func1
         0     0% 99.75%   599.33GB  9.73%  github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.BlockDataReadInner

Why is this needed ?

No response

Additional information

No response

badboynt1 commented 1 month ago

不是问题, close