SortPeers()'s current pinging strategy may not scale

[t-lin]

SortPeers() currently sorts using results from on-the-spot sequential pinging of peers. This approach will not scale as the number of peers grow, and it only provides instantaneous RTT results (disregarding historical performance).

Possible solutions include:

• Using some type of cache structure like pcache in the service-manager repo
  • Leave the pinging to be done by some other thread/goroutine (i.e. only ping allocators, and only allocators within some distance or with some metadata tag indicating region?)
• Somehow leveraging Prometheus information
  • Need to figure out how to populate it with peer-to-peer latency info, which may not be totally scalable, as that means the database will nee to include full-mesh info

[t-lin]

May have to wait for Multi-Tier-Cloud/service-manager#3