Support for multiple peers needs improvement

[Schoonology]

While I love the idea of a pure-Node alternative to ZeroMQ and the like, there are a number of areas that need some improvement before it can replace it for dynamic clusters with nodes coming in and out periodically:

• Support for disconnect-ing existing peers - This is required, but tricky. I played around on a fork for a little while, but it'd be a pretty substantial undertaking. The biggest hurdle is overhauling how axon does automatic reconnect: there's not currently a way to turn it off at all, let alone disable it for a specific peer.
• Per-peer elastic backoff - Currently, the retry time is per Axon socket, not per peer. One good peer and one bad peer will cause a lot of connection thrashing.
• Multiple-peer-aware connection state - Currently, the connected flag is flipped as soon as any peer accepts or disappears.
• Reconnection timers aren't cleared on destroy - This is more an issue with cleanly shutting down, which is impossible to do if a peer has ever disappeared suddenly.

I could certainly be misunderstanding the code, but tinkering in the guts confirms these concerns. Thoughts? How would y'all feel about my fixing some (or all) of these?

[tj]

Hmm I imagine a clean close via some command would be enough to signal that the peer isn't coming back. That wasn't something I needed personally but I agree it would be good to have!

I haven't checked out the code in a while now but I think the rest sounds about right as well

[thelinuxlich]

+1