Add post LB statistics

[ppebay]

In particular:

• how many shared blocks each rank has, and how many are the original ones ("local"), how many have been split, how many are coming from elsewhere:

  Rank 0: count=8: local=[16 32 48 64 80 144 ] remote: [35 67 ] split: []
  Rank 1: count=11: local=[] remote: [4 8 78 92 94 104 120 158 165 182 190 ] split: []
  Rank 2: count=9: local=[] remote: [10 15 29 41 63 99 106 127 150 ] split: []
  Rank 3: count=13: local=[131 147 163 179 ] remote: [5 37 44 96 108 111 116 149 ] split: [187(home=11) ]
  Rank 4: count=11: local=[20 36 68 84 132 164 180 ] remote: [0 12 39 89 ] split: []
  Rank 5: count=12: local=[21 53 69 101 117 133 ] remote: [19 24 45 55 83 107 ] split: []
  Rank 6: count=9: local=[166 ] remote: [3 23 28 112 122 123 138 143 ] split: []
  Rank 7: count=13: local=[103 119 135 151 167 183 ] remote: [13 57 65 128 137 185 186 ] split: []
  Rank 8: count=13: local=[136 152 168 184 ] remote: [62 77 81 105 121 145 160 177 191 ] split: []
  Rank 9: count=13: local=[153 169 ] remote: [1 56 60 71 97 113 129 148 170 181 ] split: [142(home=14) ]
  Rank 10: count=10: local=[26 42 58 74 154 ] remote: [76 87 176 ] split: [59(home=11) 187(home=11) ]
  Rank 11: count=10: local=[11 43 75 91 155 171 ] remote: [25 72 73 ] split: [139(home=11) ]
  Rank 12: count=13: local=[124 140 156 172 188 ] remote: [2 17 27 33 47 49 100 161 ] split: []
  Rank 13: count=13: local=[93 109 125 141 157 173 189 ] remote: [52 85 88 90 ] split: [59(home=11) 139(home=11) ]
  Rank 14: count=10: local=[14 30 46 110 126 174 ] remote: [7 9 61 ] split: [142(home=14) ]
  Rank 15: count=8: local=[31 79 95 159 175 ] remote: [40 51 115 ] split: []

• we would need to digest the above via some TBD global statistic(s) that would measure the "dispersion" of remote blocks

[ppebay]

Adding the number of shared blocks as allowed rank QOI, with toy memory problem:

[ppebay]

And with the "few" challenging problem:

[ppebay]

Also added the number of "uprooted" shared blocks QOI (i.e. those that are present on a rank but where not initially homed there):

[ppebay]

And in contrast, the number of "homed" blocks:

[ppebay]

We can also look at the above in terms of ratio (or fraction) of homed blocks relative to the total number of blocks:

[ppebay]

The same as above for the "toy" problem; unsurprisingly it was able to better retain original homing than for the more challenging problem:

[ppebay]

So @lifflander is it the kind of QOI that would fit the bill and provide the kind of post-LB information that we would then want to summarize with a global statistic?

[nlslatt]

@PPP I like these QOIs.

For the challenging example visualized above, have you printed the details like in the very first comment at the top of the page to confirm that almost nothing is still at home at iteration 8/9? That seems pretty extreme.

If the visualization is correct, it seems like there is room for improvement in the load balancer as I would expect better performance by keeping the new groupings of objects that the load balancer found but changing on which rank the entire grouping will be processed so that there might be at least one object at home on more ranks.