Super nodes cause poor join performance due to partition skew

[joshdevins]

When there are super nodes in the graph with hundreds of thousands or millions of out-edges, the PageRank join operation performs very poorly (stragglers) with skewed partitions.

[joshdevins]

Super nodes in this case might actually not be very big -- like 100k out edges even. Consider a broadcast join (although this can be huge) or a "duplicate" join, where the vertices dataset is artificially duplicated to add randomness and allow for uniformly distributed partitions.

[joshdevins]

Ideas:

• play with different partitioners and see if we can get better partitions
• broadcast join
• "duplicate" join approach (one description is: https://datarus.wordpress.com/2015/05/04/fighting-the-skew-in-spark/)