Evaluate integration of the web relay

[drwasho]

Overview

Some time ago we put together https://github.com/OpenBazaar/webrelay, which is a server designed to:

1. Publish offline messages on behalf of other nodes, specifically light clients on web and mobile
2. Forward any offline messages published to a node over sockets

This is not to be confused with the IPFS relay that facilitates two online nodes forming a connection for direct interactions. The purpose of the web relay is to make publishing offline messages more reliable, and the detection of offline messages more immediate.

This infrastructure is presently used by Zokos to great effect (kudos @rodkeys). Presently, we've started experimenting how this would work in Haven, and since things are getting more serious, we should capture the work in this issue.

Hypothesis

Our expected utility for this work:

1. More reliable publishing of offline messages (i.e. fewer missing order messages, faster offline message delivery)
2. Near-instant detection of received offline messages addressed to the node (e.g. chat, order messages)

Phase 1: Feasibility study

We want to evaluate whether our hypothesis :

• [x] 1. Deploy the web relay server
• [x] 2. Create a branch where a socket connection is being made to the web relay server
• [x] 3. Test sending/receiving offline messages using the web relay server
  • This includes desktop-desktop, desktop-mobille, desktop-zokos, mobile-zokos, and mobile-mobile messaging

The manner of this implementation will suit quick testing and not production-level code. If our testing demonstates an improvement in offline message publishing and delivery, then we'll move onto phase 2.

Phase 2: Production integration

After looking at the code we prepared in the branch, we will:

• [x] 1. Schedule a call to walk through the code and how it works
• [x] 2. Write up some concrete requirements
• [ ] 3. Come to a consensus on how we want to implement this, and any other considerations
• [ ] 4. Schedule the development work
• [ ] 5. Test and review the code
• [ ] 6. Merge and deploy

A parallel discussion needs to happen with the infrastructure team, who will be responsible for maintaining the web relay.

Other considerations

• The web relay is looking promising thus far and could significantly improve the reliability of messaging in OpenBazaar
• My personal suggestion is during the planning parts of phase 2, that we identify the minimal work required to deploy this feature to our users, and outline additional non-critical work we may want to do to enhance the system in a separate issue for future prioritisation

Requirements discussed in planning

Client

• Better name for "web relay" which disambiguates from existing IPNS circuit relay functionality.
• Send the message the fastest way possible while ensuring the mechanism doesn't break anything. One of two details enable this requirement:
  • Order message application needs to be idempotent (more broadly than offline messages being deduplicated on receipt)
  • Some assurance that keeps the potential for message duplicates isolated to offline messages (which can be mitigated by existing deduplication logic)
• Web relay only communicates with a single remote server to avoid the scope of sharing internal state across multiple app instances.
• Web relay usage is opt-out by the user via removal of configuration setting (There are privacy concerns to consider here.)
• If a recipient node is unavailable via web relay, the normal offline message path continues to work as before. (This may need to be provided by an affirmative NACK or error from the server to the client.)
• Ambiguity between OFFLINE (via DHT) and INDIRECT (via web relay) should be clear as part of the naming.
• The client needs to clearly indicate when it is failing to perform its normal function so managed failover can occur.

Server

• Web Relay is either stateless or must be a single instance (so that connection pools are not segreated by which server they are connected to).
• Highly-available deployment which allows the app to be renewed without manual intervention.
• Configurable exposed port for inbound connections

[drwasho]

Update:

1. We've done some testing of PR #1824 and it looks like the relay code is working well on desktop and mobile
2. As a bonus, Brian refactored the code so that we send the relay message first before going through the usual stack (direct, DHT, pubsub etc)
3. Brian also fixed an issue where the relay wasn't publishing the offline messages it received to the network... so now it is
4. @hoffmabc is going to add some logging to track when the relay message has been handed over to the server + when an offline message has been successfully published to the network

(3) is really important since while the relay code will make our offline messaging system more robust generally speaking, it might resolve the issue where offline chat and order messages being published to the network are being interrupted when the user hard closes the app.

Testing

• Some time ago we configured the servers to accept offline relay messaging.
• In my own testing of PR #1824, sending messages from Zokos to a relay-enabled mobile server, there isn't an improvement in the speed of detecting messages since they're already immediately detected due to the previous point
• Testing the detection of offline messages between two mobile nodes is tricky too since we have the IPFS relay that makes it difficult for two nodes to appear offline to each other (this is a good thing)... as a result, we can expect that if there is any problem with the IPFS relay, the web relay will prevent degraded performance (and if both go down, then users will only detect offline messages the next time they reopen the app, or keep it open for >10 min)
• The last test I want to run is to see what improvement is made in the delivery of offline messages to the relay vs publishing to the network

[placer14]

I think this POC could be merged for general use with the following improvements (for considering during our planning call):

• This should be an opt-in feature, enabled by the presence of the relay addresses in the config. Likewise, should the configuration for this value be unprovided, empty, or invalid; the WebRelay should not run. Additionally, an empty key or invalid data should also include a warning in the logs indicating the values were bad and the WebRelay will not run.
• Relay connections should be manageable by the caller. (Connections appear to start within a goroutine which need to also be terminable by the calling process.)
• Proper error handling throughout
• Unit tests protecting ConnectToRelay and SendRelayMessage
• Find a less ambiguous name for the package so it doesn't get confused with IPFS relay
• Simple documentation (function or package-level godocs are sufficient) which describes the purpose and usage of the behavior.

[drwasho]

@hoffmabc is going to add some logging to track when the relay message has been handed over to the server + when an offline message has been successfully published to the network

@hoffmabc we got distracted with a bunch of work obviously, but this would still be helpful to put some data behind the theory.

[placer14]

I've updated the description with the requirements as discussed in our meeting today (with a few amendments). Please ACK this comment or respond with recommended changes once you've had a chance to review.