We currently only enable unified memory manager for Comet if off-heap mode is enabled. It causes some memory issues like #996, #949, etc.
Re-thinking about this, I think a better approach is to always use unified memory management for Comet, i.e. off-heap mode must be enabled for Comet. It could simplify memory management in Comet. It can give a more explicit signal to users that off-heap memory is used for Comet native execution that should give users a more clear picture about how Spark/Comet memory is used (so they don't need to guess).
What is the problem the feature request solves?
We currently only enable unified memory manager for Comet if off-heap mode is enabled. It causes some memory issues like #996, #949, etc.
Re-thinking about this, I think a better approach is to always use unified memory management for Comet, i.e. off-heap mode must be enabled for Comet. It could simplify memory management in Comet. It can give a more explicit signal to users that off-heap memory is used for Comet native execution that should give users a more clear picture about how Spark/Comet memory is used (so they don't need to guess).
Describe the potential solution
No response
Additional context
No response