Feature Request: Add MQTT server

[sharukins]

For DIY projects with popular Microcontrollers like the ESP 8266 and ESP32 MQTT is pretty popular. Therefore implementing a MQTT-server like Mosquitto would be really nice.

[vfil]

Would be mosca.io or mqttjs an option?

[sharukins]

Sure 👍 Looks like mosca.io is better documented than mqttjs, so the former is probably better suited.

[daenney]

Wouldn't it be possible to have those devices post Web Things API compatible messages on an MQTT broker, and have an MQTT adapter in the gateway that spawns devices for it (and relays updates/actions)? Then anything that wanted to integrate with gateway could just build on Web Things + MQTT support.

Not saying it can't also ship with an MQTT server, but it seems you could get more for free if you could use the gateway as a Web Things to/over MQTT relay.

[apiel]

I am completely agree with @daenney , but right now they seem to be against to have devices client of a server. With the current Wot specification, each devices (called "thing") must be server and the gateway client to them :-/

[daenney]

That's too bad. I've taken a different approach for my own home gateway system, which centers around devices announcing and behaving using a pre-defined protoco over an MQTT broker and then automatically spawning HomeKit devices for them. We built a gateway adapter that bridges that into the WoT gateway, works really well and makes it really low effort to get all our devices in, since the only thing I need to do is map our protocol to what WoT expects. We'll probably change our internal format to be Web Things compatible once the spec settles down a little.

[apiel]

https://github.com/mozilla-iot/wot/issues/50#event-1462906766

Actually I don't even understand why they put effort in writing a middle-ware, there is already so much open source project already existing, so it would be better that they implement plugin for those project... (in another way it's cool to have solution written in JS)

I guess we can still somehow try to follow the Wot recommendation using Mqtt or what's ever protocol and provide some example for other developer. Soon or later, they will have to admit that they can't skip such use-case.

I just don't really understand why the Wot standard is so limited to their architecture :-/

[blackfyre]

How about an MQTT connection extension, if they are so against integrating a server?

[apiel]

Hi @blackfyre , after my comment, they reply on my other thread and clarify the things :p For the moment they prefer to focus on the gateway solution due to all the device not having direct internet connection, it's often the case with bulb and low energy sensors. Hopefully, in the future they will provide the other architecture. Also we can always make a solution proposal ;-)

[seoeaa]

Mqtt👍

[benfrancis]

@apiel wrote:

I am completely agree with @daenney , but right now they seem to be against to have devices client of a server. With the current Wot specification, each devices (called "thing") must be server and the gateway client to them :-/

I just don't really understand why the Wot standard is so limited to their architecture :-/

This is not true at all, I'm afraid you misunderstood.

As explained in mozilla-iot/wot#50, the gateway is just one way of connecting devices to the Web of Things.

It is true that the goal of the Web of Things is to give devices URLs on the web, and therefore a web server needs to be involved at some point. Our initial initial blog post explains how there are three integration patterns for connecting devices to the Web of Things:

• Direct
• Gateway
• Cloud

The Things Gateway implements the gateway integration pattern where resource constrained devices or devices using non-web or non-Internet protocols (like Zigbee, Z-Wave and Bluetooth) can be bridged to the Web of Things. The gateway acts as the Web of Things server by exposing the Web Thing API.

The direct integration pattern is for devices which expose the Web Thing API themselves (e.g. like a WiFi camera).

The cloud integration pattern is where a device uses some other protocol (e.g. MQTT) to connect to a cloud server and the cloud server exposes a Web Thing API for the device.

Project Things will consist of three components:

• Things Gateway (our initial focus)  -  An open source implementation of a Web of Things gateway which helps bridge existing IoT devices to the web
• Things Cloud  - A collection of Mozilla-hosted cloud services to help manage a large number of IoT devices over a wide geographic area
• Things Framework  - Reusable software components to help create IoT devices which directly connect to the Web of Things

As you can see, these map onto the three integration patterns.

We definitely want to support bridging MQTT to the Web of Things, the question is what is the most useful way of doing that?

One thing we could do is to create a cloud service which provides an MQTT server and bridges MQTT to the Web Thing API. This would use the cloud integration pattern where a device could act as a client to the cloud service, and applications could interact with the device over the Web Thing API without even needing to know that it's using MQTT underneath. You would need give the cloud service information about the device so that it can generate a Thing Description, and there would have to be some mapping from MQTT messages to Web Thing API messages.

We could also create an MQTT adapter add-on for the gateway which provides an MQTT server and talks to the gateway over IPC. But what would be the use cases for this adapter? Is it actually useful? MQTT itself is just a messaging protocol, it doesn't expose any information about the capabilities of a device needed to generate a Thing Description and doesn't have defined device types or device capability types like protocols such as Zigbee and Z-Wave do. That means you either need a separate MQTT adapter per device, or every device needs to be custom-built to use a common set of messages which the gateway adapter understands.

Just running an MQTT server on the gateway doesn't achieve anything with respect to the Web of Things. We need to better understand the use cases people have for MQTT and how best to support those use cases using one of the three integration patterns.

[sharukins]

okay, I see it would probably be more reasonable to have a web of things implementation for ESP32 and 8266.

[benfrancis]

I guess we can still somehow try to follow the Wot recommendation using Mqtt

Note that MQTT is not a web protocol. It might be possible to use the same message types over MQTT as for the Web Thing WebSocket API (we actually use almost the same format internally on the gateway over IPC too), but that won't be the Web of Things. MQTT can be used as an Internet of Things protocol, but will always require a bridge/gateway to connect it to the Web of Things using a web protocol.

The Web of Things is designed to create a unified application layer on top of IoT protocols like MQTT, where things have URLs and use a common data model and API.

[benfrancis]

okay, I see it would probably be more reasonable to have a web of things implementation for ESP32 and 8266.

Exactly. Our team has just started working on the device framework for existing IoT platforms like Arduino and Android Things to make it easy to build native web things which directly expose the Web Thing API over an IP network.

If you're building a device which is capable of running a full TCP/IP stack and HTTP/WebSockets server then you should consider making it a "native web thing" which serves its own Thing Description and API. If you have a more constrained device you might want to consider using the gateway integration pattern to bridge it to the Internet/web.

[apiel]

In some way MQTT support would still be good even for Low energy device. In my previous job, we was using MQTT over Zigbee, it was then connecting to our gateway to route all the message with RabbitMQ. In some way MQTT is very common in Iot and I guess it would be good to try to somehow unify as much as possible the messages sent over or not Web socket with or without MQTT protocol. JSON message can be sent over WS and MQTT so there is no bottle neck there. The only thing to clarify, is how each device can be recognize by the server/gateway, and this is currently missing in your Wot specification unless we use REST. You are actually very close to be able to support such use-case, so it would just be cool to have :p

Anyway thanks for your work ;-)

[seoeaa]

In iobroker, MQTT automatically adds all the devices to the list and at least this happens with my sonnof devices with Sonoff-Tasmota firmware. What for a cloud ??? The system should work without the Internet. I'm against the cloud this is a bad approach.

[benfrancis]

@apiel wrote:

The only thing to clarify, is how each device can be recognize by the server/gateway, and this is currently missing in your Wot specification unless we use REST.

What is missing? To be a web thing a device needs to provide a web thing description over a web protocol. In practice that usually means HTTP. It could be CoAP which is very similar to HTTP. A WebSocket is just an upgraded HTTP connection.

[apiel]

As I say you are very close to it. https://iot.mozilla.org/wot/#x3-1-thing-resource

In this description, it is tight to REST. The gateway make a request to the device to get the description.

So there is few stuff that are too specific to your architecture:

GET http://mythingserver.com/things/pi
Accept: application/json

and

"href": "/things/pi/properties/temperature"

Would be great to try to make all this communication "protocol agnostic" and this actually a bit tricky ;-)

[seoeaa]

Why do you need this wot if you can not do mqtt.

[benfrancis]

Would be great to try to make all this communication protocol agnostic and this actually a bit tricky ;-)

Retrieving a Web Thing Description by its URL should work for any web protocol. MQTT is not a web protocol, it requires a gateway or cloud service to act as a web server and bridge it to the Web of Things. The web server can serve the web thing description.

[benfrancis]

Just to make this discussion even more interesting, one thing that is quite common is using MQTT over WebSockets. AWS IoT and Azure IoT both provide APIs using MQTT over WebSockets.

This would not be compliant with our proposed webthing WebSocket subprotocol as is acts as a completely separate WebSocket sub-protocol. It could be possible to use the same message formats we're proposing but wrapped in an MQTT message in order to benefit from all the QoS features of MQTT (I believe this is something that EVRYTHNG do). But that does complicate things somewhat because the client and the server need to somehow negotiate whether they're using the webthing sub-protocol or the mqtt sub-protocol with a webthing sub-sub-protocol!

This is probably the opposite way around than what people here are suggesting though, because the web browser would probably be the MQTT client and the "thing" would be the server.

[rei-vilo]

Interesting thread. The major requirement with MQTT is its reliance on a TCP/IP stack.

For BLE, ZigBee et al., there is the MQTT-SN variant, SN for Sensor Networks.

MQTT-SN v1.2, formerly known as MQTT-S, is available here. MQTT for Sensor Networks is aimed at embedded devices on non-TCP/IP networks, such as Zigbee. MQTT-SN is a publish/subscribe messaging protocol for wireless sensor networks (WSN), with the aim of extending the MQTT protocol beyond the reach of TCP/IP infrastructure for Sensor and Actuator solutions.

• MQTT For Sensor Networks (MQTT-SN) — Protocol Specification — Version 1.2

[benfrancis]

I wonder whether this could be separated out into an MQTT server and then MQTT add-on adapters which add support for a given MQTT device or group of devices to that server. So the core gateway image bundles an MQTT server and then add-ons use that server, so that every add-on doesn't have to bundle its own server.

But what would be the interface for add-ons to authenticate with the MQTT server and register topics that can PUBLISHed to?

[hobbyquaker]

My 2 Cent: I think a MQTT adapter add-on for the gateway would be the best approach. No need to integrate a Broker imho, people that wanna use MQTT most likely already have the necessary infrastructure running. As many MQTT based architectures don't even publish device descriptions and there is no widely adopted standard for that on MQTT (e.g. Homie tries to standardize that, but I wouldn't call that "widely adopted") I think it will be necessary to have the possibility to manually configure the devices / Thing Descriptions and the mappings between thing properties and MQTT topics. The addon should also be able to parse JSON payloads (as this is what imho most MQTT based systems use) with configurable properties that are then used for the mapping to things properties. The whole approach would be quite similar to homekit2mqtt, a software that creates HomeKit devices with configurable mappings to MQTT Topics/Payloads.

[mrstegeman]

This should probably just be a part of individual adapters, rather than central to the gateway. If someone has a strong argument to the contrary, feel free to reopen.