Open viirya opened 1 month ago
We are going to disable Comet shuffle if AQE coalesce partitions is enabled due to potential memory leak issue on Java Arrow StreamReader (#380).
We should address this issue and fix it to enable Comet shuffle with AQE coalesce partitions.
No response
What is the problem the feature request solves?
We are going to disable Comet shuffle if AQE coalesce partitions is enabled due to potential memory leak issue on Java Arrow StreamReader (#380).
We should address this issue and fix it to enable Comet shuffle with AQE coalesce partitions.
Describe the potential solution
No response
Additional context
No response