Optimize mapJoinPartitions, change RDD2 into shuffle dependency.
So that RDD2 partitions won't be re-serialization and re-computation when iterating.
Keep old codes and add a new class MapJoinPartitionsRDDV2, and add java option vflbfgs.mapJoinPartitions.shuffleRdd2, when this option set true, will use new implementation, else use the old one.
Optimize mapJoinPartitions, change
RDD2into shuffle dependency. So thatRDD2partitions won't be re-serialization and re-computation when iterating.Keep old codes and add a new class
MapJoinPartitionsRDDV2, and add java optionvflbfgs.mapJoinPartitions.shuffleRdd2, when this option settrue, will use new implementation, else use the old one.