As described in detail in https://github.com/apache/datafusion-comet/issues/949, when using native memory management, we currently create one pool per plan and do not take executor concurrency into account when setting spark.executor.memoryOverhead.
Describe the potential solution
This issue is for implementing a shared pool between native plans within an executor.
What is the problem the feature request solves?
As described in detail in https://github.com/apache/datafusion-comet/issues/949, when using native memory management, we currently create one pool per plan and do not take executor concurrency into account when setting
spark.executor.memoryOverhead.Describe the potential solution
This issue is for implementing a shared pool between native plans within an executor.
Additional context
No response