feat: adding redundancy

[sameh-farouk]

what's changed

• [x] Core:

  • [X] SubstarteTwinDB can read and update chain with multiple relay domain. (note: I used _ as a separator because it is workable without requiring any blockchain side changes. It is the only character accepted by tfgrid pallet relay validation that not permitted in domain names in accordance with RFC 1035, which only allows letters, digits and hyphens. So using it as a separator makes sense ATM till further planning or pallet modification if needed.
  • [X] rmb-peer can maintain parallel websocket connections to multiple relays
  • [X] Read from all sockets
  • [X] Write to only one (try one by one till succeeded for now)
  • [X] Remove federation field usage from rmb-peer and relay and update federation logic to happens in the relays
  • [x] Adjust protobuf message

• [x] Performance optimization:

  • [X] On rmb-peer, reorder sockets to list the last known working socket first.
  • [X] On relay, implement relay ranking system. I tried to avoid storing a lot of info or using complex algorithm. The system has a simple interface consisting of two methods: downvote for reporting misbehaving service and reorder that takes a twin relay list and sorts them in place according to their known mean failure rate over a recent configured period of time. This gives a hint to the router to try the relays that have a higher chance of working first, to minimize routing messages to services that failed recently.

  The rank will heal over time because the system will only consider failures in the recent time frame, allowing the router to revisit the relay and distribute the load between all working services. For relays with the same failure rate, the order will be randomize.

• [x] Adjust and fix federation tests according to changes above:

Related Issues:

• https://github.com/threefoldtech/rmb-rs/issues/141

[sameh-farouk]

Good job Sameh, I wouldn't done any better.

Thanks @muhamadazmy . You truly made my day!