Messaging HA setup

[ambertch]

Add a second messaging server and create an HA setup. Some considerations may include:

• the dapp connects to the messaging server over websocket. What would be the effect of a client connecting to one server, then another (ex. with HTTP, a mobile phone switches from wifi to cell network, and the change in IP causes requests to go to another server)? Is this an issue with websockets / does the setup require sticky sessions?
• how do the two servers sync data between one another? OrbitDB contains a CRDT, although are there any conditions under which there could be data loss?

[ambertch]

Having the domain name messaging.originprotocol.com resolve to multiple IPs will not work:

• Client A connects to one server, then another server
• Client B sends a message to A, and it goes to the first server that A connected to
• The message will not be delivered.

An option is to use an array of persistence servers and implement keep-alive + reconnect logic on the messaging client. Then, if a client cannot reach a server it can connect to another one.

A corner case, albeit a small likelihood, might be if a client cannot reach a server that is not actually down.

[franckc]

Initially I think it's probably ok if we have a single OrbitDB server, as long as we have monitoring and that we detect quickly if the server is down and that we either automatically or manually spin up a new one.

I think the most import goal in terms of messaging system reliability is to make sure that there is no message loss under any circumstances. If we show the user the message was sent, it should never get lost. Perhaps we can think about how we can provide this guarantee under the different failure scenarios ? In particular:

• OrbitDB daemon or host crash
• IPFS server or host crash