apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.16k stars 421 forks source link

[VL] Limit Velox global memory manager's usage #6960

Open zhztheplayer opened 1 month ago

zhztheplayer commented 1 month ago

It's observed that Velox backend uses more memory than we configured. Which is perhaps related to the untracked Velox global memory manager.

We should set a capacity according to Spark overhead size to here to limit that memory manager's usage.

FelixYBW commented 1 month ago

The global memory allocation is used for spill's compression buffer. The memory is counted into executor.overhead memory in Spark. We can do the same for Gluten.

In spark, the overhead memory includes reducer's netty memory, compression buffer memory, the memory native library allocated.

In Gluten the overhead memory includes reducer's netty memory, memory using global allocator, all the std::container's memory.

FelixYBW commented 1 month ago

@marin-ma Is the compression buffer retained in shuffle? if not we can use the global memory allocator to allocate it.