When running large data task，spark driver offen encounters oom error. By analyzing the heap dump，I found there are 185,657,950 PartitionInfo instances that occupy 4GB memory. Most PartitionInfo instances are referenced by ShuffleTaskManager.shuffleEpochSetMap attribute. This attribute type is ConcurrentHashMap[String, ConcurrentHashMap[Int, util.List[PartitionInfo]]]. It's only used to check shuffle data loss.
We can change List to Set, remove duplicate PartitionInfo. After doing this, the memory is used much less than before.
What type of PR is this?
enhancements：reduce driver memory usage
What this PR does / why we need it:
When running large data task，spark driver offen encounters oom error. By analyzing the heap dump，I found there are 185,657,950 PartitionInfo instances that occupy 4GB memory. Most PartitionInfo instances are referenced by ShuffleTaskManager.shuffleEpochSetMap attribute. This attribute type is ConcurrentHashMap[String, ConcurrentHashMap[Int, util.List[PartitionInfo]]]. It's only used to check shuffle data loss.
We can change List to Set, remove duplicate PartitionInfo. After doing this, the memory is used much less than before.