Add support for devices raising alert messages

[dejanb]

We want to have a support for a new kind of message with even higher priority than events. We want to separate these messages, from regular events and create a new "alert" address for them. They should have the same QoS as events (AT LEAST ONCE). For example, this channel can be used to send messages about device malfunctions, which is separated from the important events device generates.

[pellmann]

In addition, per default the messages should be persisted. And the proposed address is alert/tenant.

[sophokles73]

We could either create a completely new API analogous to the Event API or we could extend the existing Event API with the alert endpoint and simply describe its purpose to provide an additional level of (message) priority that the client can choose from.

Given that the implementation and delivery semantics are actually the same as in the Event API, I would rather go for the latter option. Any thoughts, @pellmann @dejanb @ppatierno @sysexcontrol ?

[pellmann]

Does anyone know if the QPID Dispatch Router supports the priority header of messages? From an API level we could define a higher priority for alert events or we could argue that this header could be used for different events in a fashion, that the project (this tenant) needs. So we do not need to change or add anything to support the requirement of higher priorities. WDYT @sophokles73 @dejanb @ppatierno @sysexcontrol ?

[sophokles73]

Does anyone know if the QPID Dispatch Router supports the priority header of messages?

My understanding is that the Dispatch Router doesn't buffer messages, so if two subsequent messages with different priorities arrive, e.g. message one with prio 7 and message two with prio 2, then the router will try to deliver message one to a consumer and will release it, if no consumer is available. When message two arrives, the same processing happens. Thus, the priority has no meaning to the dispatch router itself but may be relevant for how a broker that the messages are routed to ..

[kaniyan]

When we talk about connecting large numbers of IoT devices to Hono, IMHO a feature such as this is required. There by the devices are able to report critical incidents in a higher priority manner and that enables north bound side to act up on those incidents swiftly.

Does anyone know if the QPID Dispatch Router supports the priority header of messages?

My understanding is that the Dispatch Router doesn't buffer messages, so if two subsequent messages with different priorities arrive, e.g. message one with prio 7 and message two with prio 2, then the router will try to deliver message one to a consumer and will release it, if no consumer is available. When message two arrives, the same processing happens. Thus, the priority has no meaning to the dispatch router itself but may be relevant for how a broker that the messages are routed to ..

I ran a simple test by sending event messages with priorities varying from 0 to 9. After the messages are sent, a consumer (north bound side) is started up. The consumer received those event messages in accordance with the message priority i.e the messages with priority 9 arrived first and at last the messages with priority 0. It is also mentioned in the Apache ActiveMQ Artemis documentation, that the the broker will attempt to deliver higher priority messages before lower priority ones. In addition to that, Qpid dispatch router docs describes that the higher-priority messages will travel more quickly through the router network than they otherwise would.

Extending the existing Event API with the alert endpoint:

We could either create a completely new API analogous to the Event API or we could extend the existing Event API with the alert endpoint and simply describe its purpose to provide an additional level of (message) priority that the client can choose from.

@sophokles73 @dejanb My understanding regarding the above option is that the devices would send the alert messages to a new alert endpoint. Then the protocol adapters would treat them as event messages with higher priority (by setting higher priority for messages) and forward them using the same AMQP address event/${tenant_id}. A northbound application using hono client library would consume those messages as events (by using createEventConsumer(...) in ApplicationClientFactory class). Please correct me if my understanding is different.

I understood that in this approach, the alert and the regular event messages are forwarded using the same tenant specific AMQP address (event/${tenant_id}). In a system with large number of devices under a tenant, the alert messages sent by many devices could impact the delivery of the regular event messages and especially depending upon how quick the messages are consumed by the north bound side. The impact could be longer delays in delivery of the regular event messages or no delivery if events with short ttl duration are delayed. In order to avoid it, FMPOV it makes sense to use a new address such as alert/${tenant_id} and send the alert messages. Also dispatch router already has a provision to set the priority at the address level as given in the below example. Qpid dispatch router docs describes that this priority is to be assigned to all messages sent to this address.

address {
    prefix: alert/
    priority: 9
    ...
}

Or as you mentioned, choosing the priority can also be left to the client. While sending alert messages, devices may set an extra header or a property-bag for the message priority to be used by the protocol adapters.

Since this approach talks about new alert endpoint to be used by the devices for alert messages, IMHO it also makes sense to use a new API at the north bound side to receive those messages something like createAlertConsumer(...) in ApplicationClientFactory. Thereby the north bound side can act upon those alerts differently without any extra effort to identify if an event message is an alert message or not and this may be done by checking the message priority or any headers.

WDYT?

[sophokles73]

There by the devices are able to report critical incidents in a higher priority manner and that enables north bound side to act up on those incidents swiftly.

We haven't received any feedback yet from users who would like to see alerts in addition to the already existing events. Thus, we didn't really follow up on the idea so far. Do you have a particular use case where this is required?

I can imagine though, allowing devices to indicate a priority for events they publish using a header/property.

[kaniyan]

There by the devices are able to report critical incidents in a higher priority manner and that enables north bound side to act up on those incidents swiftly.

We haven't received any feedback yet from users who would like to see alerts in addition to the already existing events. Thus, we didn't really follow up on the idea so far. Do you have a particular use case where this is required?

The use case I think of is how quickly critical incidents can be routed with high priority. For example, it could be a device reporting a power failure in some systems or reporting a temperature increase to critical levels. Currently in Hono, such incidents can be reported as events and are delivered in a normal pace as any other events. As a user, I need to act upon such events immediately than the other regular events. Instead of alerts, allowing devices to indicate a priority for events could be a solution. But it comes at it own cost which is: high priority events of some devices could disrupt the delivery of normal priority events from other devices (as explained in my previous comment).

I can imagine though, allowing devices to indicate a priority for events they publish using a header/property.

Hono already support this feature partially i.e the devices that communicate via AMQP Adapter can set the priority directly on the message. IMHO this should also be extended to other adapters.

[sophokles73]

high priority events of some devices could disrupt the delivery of normal priority events from other devices (as explained in my previous comment).

I guess the assumption here would be that alerts are even less frequent than (normal) events. Thus, IMHO this shouldn't be a problem ...

IMHO this should also be extended to other adapters.

Then maybe this would already be sufficient? At least it would give the devices more options than today. BTW, the fact an AMQP device can set the priority is an undocumented (and also unintended) feature, I would say ;-)

[kaniyan]

Then maybe this would already be sufficient? At least it would give the devices more options than today.

:+1: I think so too. The critical incidents can then be sent as events with high priority. I would like to do a PR for this.