[Access] Draft design of new WebSockets

[Guitarheroua]

User Story: WebSocket Subscription Management

1. The client establishes a single WebSocket connection with the Access Node (AN) through, for example, ws://localhost:8080/ws and maintains this connection until closed by the AN or the client itself.

2. To subscribe to necessary topics, the client sends a special message through the WebSocket connection:

   ws.send(JSON.stringify({
    action: 'subscribe',
    topic: 'events_from_latest'
   }));

   The client can subscribe to multiple topics through a single connection but must manage messages received.

   QUESTION: Should we have the possibility to subscribe for multiple topics with a single message, providing subscriptions as an array?

   If necessary, the client could pass initial arguments to the subscription, by adding them to the arguments object field:

   ws.send(JSON.stringify({
    action: 'subscribe',
    topic: 'events_from_start_height'
    arguments: {
        height: '123456789'
    }
   }));

   The response from the AN containing updates will resemble the following:

   ws.onmessage = (event) => {
    const message = JSON.parse(event.data);
   
    /*
    The JSON message will appear in next format:
   
    {
      topic: 'events_from_latest',
      data: [...]
    }
    */
   
    switch (message.topic) {
      case 'events_from_latest':
        // Handle events
        break;
      default:
        console.log('Received publication for unsupported topic:', message.topic);
        break;
    }
   };

3. To unsubscribe from a topic, the client sends another message through the WebSocket connection:

   ws.send(JSON.stringify({
    action: 'unsubscribe',
    topic: 'events_from_latest',
   }));

4. To get the list of all active subscriptions, the client sends the next message through the WebSocket connection:

   ws.send(JSON.stringify({
    action: 'list_subscriptions'
   }));

5. The client can close the connection when it is no longer needed or by the AN if the client has unsubscribed from all topics. Note: if the client closes the connection, all subscriptions will be lost on the AN side.

Access Node Implementation requirements

1. The router.go

The router should maintain both - the new and old WebSocket (WS) connections for backward compatibility, with plans to deprecate the old WS at some point. The old WS connection will handle only one endpoint:

1. subscribe_events

The new WS connection will add support for the following subscriptions:

1. events_from_start_block_id
2. events_from_start_height
3. events_from_latest
4. account_statuses_from_start_block_id
5. account_statuses_from_start_height
6. account_statuses_from_latest_block
7. blocks_from_start_block_id
8. blocks_from_start_height
9. blocks_from_latest
10. block_headers_from_start_block_id
11. block_headers_from_start_height
12. block_headers_from_latest
13. block_digests_from_start_block_id
14. block_digests_from_start_height
15. block_digests_from_latest
16. transaction_statuses

However, these will not be individual endpoints for the router. Instead, they should be handled within the new WebSocketBroker as message subscription topics. The endpoint route for all subscriptions should be ws.

2. The WebSocketBroker

WebSocketBroker is a module similar to the old WebsocketController, but with a few major changes:

1. The broker will open a connection immediately during its construction phase but will not subscribe to any topic. Instead, it will add error handling and ping/pong service messages to track the connection.

2. The broker will keep track of active subscriptions under certain topics. This can be represented as a map, where the topic is the key. This way, it maintains control and ensures it does not subscribe multiple times to the same topic under the same connection. It can also easily unsubscribe from any topic. Examples of topics include events_from_latest, blocks_from_start_block_id, etc. This structure also allows for the easy addition of new topics for future subscriptions.

3. The broker will listen for messages from the client, waiting for the subscribe or unsubscribe actions specified in the JSON message object. Upon receiving such a message, it will call the respective methods to either create a subscription for the connection under the selected topic or close the subscription for that topic.

4. When a subscription is created for the first time in the subscribe method, the broker should bind it to the respective handler to form the correct response for the client, including the topic and data. The client will rely on the topic to parse the streamed data.

5. In the current implementation of WebsocketController, there is only one endpoint handler -WebsocketController::writeEvents which is hardcoded to WebsocketController. This should be changed to support multiple topics, with the handlers selected based on the topic from a constant map. This map should be easily expandable with new handlers for new topics and should not affect the WebSocketBroker implementation when doing so. For example:

   []topichandlers{
   "events_from_latest": EventFromLatest,
   "blocks_from_start_block_id": BlocksFromStartBlockIDHandler,
   ...
   }

6. The broker should handle connectivity, including managing ping/pong messages and handling unsubscriptions. If the client does not respond to service messages or unsubscribes from all topics, the broker should gracefully close the connection.

A visual representation of the new REST subscription process:

New Pub/Sub API Description

1. The router.go

A new AddWsPubSubRoute function will configure the route for the new subscription mechanism, using a different address than the current v1/subscribe_events route, such as v1/ws. There should only be one main route and one main handler for the new pub/sub mechanism. Different topics (similar to routes in REST) will be handled by the WebSocketBroker, reacting to messages received from the client.

2. The WebSocketBroker

The WebSocketBroker will manage subscriptions within a single connection between the client and the node.

type WebSocketBroker struct {
    conn      *websocket.Conn
    subs      map[string][string]SubscriptionHandler // where the first key will be the topic and the second one is subscription ID
    broadcast chan []byte
    /*
    ...
    Other fields are similar to WebsocketController, except for `api`,
   `eventFilterConfig`, and `heartbeatInterval`, which are specific to event streaming.
    */
}

The conn field represents the WebSocket connection used for bidirectional communication with the client. It will handle incoming messages from the client and broadcast messages to the client based on subscribed topics. It will also handle ping/pong service messages, error management, and connectivity issues.

The methods associated with the conn field are:

1. readMessages: This method will be called once and will run as long as the connection is active. It will be responsible for retrieving, validating, and processing messages from the client. The actions it handles include subscribe, unsubscribe, and list_subscriptions. Additional actions can be added as needed.
2. broadcastMessages: This method will also be called once and will run as long as the connection is active. It listens on the broadcast channel, retrieves responses from all subscriptions or methods writing to the channel, and sends responses to the client.
3. pingPongHandler: This method will be responsible for periodically checking the connection’s availability by handling ping/pong messages. It will run as long as the connection is active.

The methods associated with the subs field are:

1. subscribe: This method will be called by readMessages when the action field in the client message is subscribe. It takes the topic from the message’s topic field, creates, using some factory, the appropriate SubscribtionHandler for the topic and adds it to the subs map. It will notify the client that the subscription for the requested topic has been successfully created under the specific ID.
2. unsubscribe: This method will be called by readMessages when the action field in the client message is unsubscribe. It will take the topic and the subscription ID from the message and call related SubscriptionHandler::CloseSubscription and remove the handler from subs map.
3. listSubscriptions: This method will be called by readMessages when the action field in the client message is list_subscriptions. It will gather all active subscriptions for the current connection, format the response, and notify the client.

The SubscriptionHandler

type SubscriptionHandler interface {
   CreateSubscription(topic string, arguments []string, broadcast chan []byte) string, error
   CloseSubscription() error
   //messagesHandler should be private? 
}

The SubscriptionHandler will be an interface that abstracts the use of actual subscriptions in the WebSocketBroker. Concrete SubscriptionHandler implementations will be created during the WebSocketBroker::subscribe call, based on the topic field specified by the client. For example, topics like events_from_start_block_id, events_from_start_height, and events_from_latest will have an EventsSubscriptionHandler implementation responsible for managing event subscriptions.

All subscriptions will be unique and have different IDs and distinct instances of SubscriptionHandler stored in the WebSocketBroker. Unique subscriptions will be created for all topics, as all of them could provide different results for the client so it should be possible to subscribe to such topics multiple times with different presets.

Each CreateSubscription should take topic, additional arguments from the arguments field specified by the client, and the WebSocketBroker::broadcast channel as arguments and store them. In this constructor method, the subscription.Subscription should be created based on the backend API where the subscription is implemented.

Each handler should have a messagesHandler method that receives messages from the node subscription and formats them in pub/sub style for the end client. The formatted messages will then be forwarded to the stored broadcast channel.

The CloseSubscription method will be responsible for gracefully shutting down the subscription.

[peterargue]

Looks great @Guitarheroua. A few comments:

• I think the subscribe/unsubscribe needs an ID field to uniquely identify a subscription on a topic that could have multiple instances (e.g. events, account statuses)
• re: “The router should maintain both - the new and old WebSocket (WS) connections for backward compatibility”
  • I think we should create the new system hosted on a new endpoint, and leave the old system in place with no interoperability between them.
  • this would be the simplest and incentivize everyone to move over.
• We may also need a type field on responses to differentiate between different response message types.
• We should include special response messages from the subscribe and unsubscribe actions returning details about the subscription (like ID)
• let's include a list_subscriptions action that returns a list of all active subscriptions

[Guitarheroua]

@peterargue What is described here is the Web Application Messaging Protocol, or simply WAMP Protocol, which has a few implementations in Go. The most popular one is NEXUS, which implements the WAMP protocol and includes the features we need. It's also actively maintained, with a new version released this year. While it seems like a good fit for our requirements, we should first discuss the pros and cons of using it. My main concern is the subscription model on the client side. To me, it adds an extra layer of complexity, and clients might not be happy with that.

[Guitarheroua]

@peterargue What is described here is the Web Application Messaging Protocol, or simply WAMP Protocol, which has a few implementations in Go. The most popular one is NEXUS, which implements the WAMP protocol and includes the features we need. It's also actively maintained, with a new version released this year. While it seems like a good fit for our requirements, we should first discuss the pros and cons of using it. My main concern is the subscription model on the client side. To me, it adds an extra layer of complexity, and clients might not be happy with that.

We agreed that the Nexus library offers many useful features, but it also includes a lot of unnecessary functionality that we won't use. Additionally, the WAMP protocol implemented by this library adds an extra layer of complexity, particularly on the client side, making it more challenging to handle.