Design of message-based abstraction layer on top of hypercore

[aschrijver]

(NOTE This showcase is part 2a of Positioning, vision and future direction of the Dat Project)

I'll presented vert.x in more detail in [part 4] of this article series, but mention it here because of its message-based system design.

Vert.x has the concepts of a distributed event bus and event bus bridges.

Officially supported toolkit modules are a TCP Eventbus Bridge, a Camel Bridge, a SockJS Proxy Service, and various community-offered bridge implementations (another SockJS impl, ZeroMQ, AMQP - Kafka bridge, Saltstack bridge, server-sent events bridge)

They've kept their message design as simple as possible.

The event bus supports 2 communication modes:

• publish / subscribe
• send / reply

On the wire protocol level everything is a Frame:

<Length: uInt32><{
   type: String,
   address: String,
   (replyAddress: String)?,
   headers: JsonObject,
   body: JsonObject
}: JsonObject>

Only the following 4 frame types exist:

• sendto send a message to an address
• publish to publish a message to an address
• register to subscribe to the messages sent or published to an address
• unregister to unsubscribe to the messages sent or published to an address

The headers can be any JSON e.g.:

• from a simple map of metadata key/value pairs
• to a JSON-Schema or even JSON-LD document semantically describing the body payload, or what have you..

The body payload can be any data type, to be determined from header metadata, need not be JSON. Could be Buffer, hex-encoded image, etc.

Some thoughts:

• The address and replyAddress could be dat://-urls (useful for handshakes)
• May want to include a signature field
• This could be easily implemented in hypercore-protocol .proto definition
• Maybe what's currently there should become Frame types
• May be wise to standardize, or advice on some preferred header formats

Finally, if Dat message format is only in the slightest way compatible to what Vert.x now has, you easily write an event bus bridge impl. for it, and Dat will now have added a whole ecosystem and dev community to their base!

--

Next part: Investigating message abstraction layer implementation options