Add More Scalable Event Mechanism(s)

[mmccool]

Currently SSE is required to be used for event notifications, but it's not very scalable, and an alternative like Web Hooks would be useful when a TDD is implemented as a cloud service.

See also Issue #417

[egekorkan]

Just make sure to know that Web Hooks implies Consumers having Server functionality so it adds another type of requirement. It would make the TDD more cloud compatible but make the Consumer implementations complicated, i.e. no browser could be a TDD consumer with Webhook events. Why not mention websockets in this case?

[farshidtz]

SSE and WebSockets aren't much different in this context. SSE on HTTP/2 is actually more scalable than WebSockets because of the multiplexing capabilities. One major problem with both is the implementation of QoS which is necessary for sleeping devices and a burden for servers.

IMO, a broker-based pub-sub protocol such as MQTT is most appropriate and widely used. See also #134.

[relu91]

I agree with what has been said above. To me, those are all valid use cases we should try to make the spec flexible enough to include MQTT-based TDDs or Webhook (rather than cloud I think webhooks should apply better in any microservice scenario).

On the other hand, this complicates the implementation of a generic consumer but this is not the first time that we introduce optional methods for similar functionalities (e.g. for search we have XPath and JSONPath).

[benfrancis]

Having looked at HTTP-based event mechanisms in great detail for WoT Profile, it isn't quite as straightforward as Webhooks being more scaleable than Server-Sent Events. There are circumstances under which Webhooks scale terribly or simply aren't possible.

Server-Sent Events:

• A W3C standard implemented in all major web browsers
• Requires a TCP connection to be kept open for each subscription
• All events for a given subscription are streamed over a single TCP connection, with a very lightweight payload format
• Web browsers have a hard limit on the number of event sources which can be kept open simultaneously, and it's a small number (other SSE clients do not have this limitation, but operating systems do have upper limits on the number of simultaneous open sockets)

Webhooks:

• A loose design pattern with no single formal standard (There's W3C WebSub which is a more complicated mechanism built on Webhooks, and various competing informal specifications)
• Requires that both Producer and Consumer are both an HTTP client and an HTTP server, which is not possible for many Consumers (e.g. browsers, mobile apps, desktop apps), only other services
• Requires a separate TCP connection per event
• May require safeguards (e.g. rate limiting or abuse protection) to prevent overwhelming Consumers (intentionally or unintentionally)
• Difficult to describe in a Thing Description due to the bi-directional nature of the HTTP requests
• Does not require keeping a TCP socket open between events

WebSockets:

• A W3C standard implemented in all major web browsers, but just a bi-directional raw socket with no semantics so requires a defined sub-protocol
• Requires a TCP connection to be kept open for each Thing (or Directory), but with a defined sub-protocol that connection could be shared by all interaction affordances (properties, actions and events)
• Currently difficult to describe in a Thing Description, relies heavily on the definition of a shared sub-protocol
• All events for a given subscription are streamed over a single TCP connection

In short, Server-Sent events works best for subscribing to high frequency events from a small number of Things whereas Webhooks work best for low frequency events sent from a large number of Things (but only if the Consumer can act as a web server). WebSockets have the benefit of being able to share a single connection between all interaction affordances of a Thing (or even multiple Things/Directories from the same origin), but still require a separate connection per origin.

Although it would certainly have scalability benefits for events, a non-web protocol like MQTT is not well suited to all of the other interaction affordances exposed by a directory. Requiring directory consumers to support both HTTP and MQTT would add significant implementation complexity. Also bear in mind that specifying multiple event mechanisms may harm interoperability, if some directories or some consumers only support a subset of the mechanisms.

[mmccool]

The current document says the following: "The rest of this section describes the implementation details on top of the SSE protocol. Realizing the notification functionality using other protocols such as MQTT are possible and may be formalized in future versions of this specification."

So - how about MQTT for eventing? This was also mentioned as an option for heartbeats, see #164

[mmccool]

One possible solution here:

• Support multiple mechanisms for eventing.
• For backward compatibility we probably want to keep SSE.
• Then need a way for a client to pick which mechanism they want - different entry points?
• May also need a mechanism for a client to check with mechanisms are available in a particular directory service - but maybe the TD can list them, so that's taken care of.
• Right now the current entry point has only SSE. However, the set of forms is an array, so we could add additional options to that array for different protocols.
• A little tricky though for protocols that need a "setup" link as well.

[lu-zero]

The current document says the following: "The rest of this section describes the implementation details on top of the SSE protocol. Realizing the notification functionality using other protocols such as MQTT are possible and may be formalized in future versions of this specification."

So - how about MQTT for eventing? This was also mentioned as an option for heartbeats, see #164

I think CoAP Observe might be simpler, but any protocol that has an observe pattern as primitive should do.