[MQTT]: Request Response

[egekorkan]

How does MQTT community solve request response style deterministic interactions? Since the topics are always one way, meaning that a publisher publishes to a topic and subscribers get it but if the subscribers publish to the same topic a response message, it will look as if the original publisher published something. Thus, do MQTT developers create another topic when they want to implement this behavior? E.g. a publisher publishes its "request" to /mytopic and subscribers respond with the "response" to the /mytopic/response ? It seems that with MQTT5 it is now possible! See here. It is seen that it is upto the publisher (request maker/Consumer) to specify this topic and it can be seen as a private information since the publisher/Consumer may not want this information to be known. How can we signal this in a form field? It is tricky since the Exposed Thing should have the feature to extract this information from the header reply back. Maybe something like "mqv:responseTopic": true }] ? If we opt in something like this, we can also have true readable properties. See also #84

[egekorkan]

We have decided that the new req-resp can be a subprotocol like "subprotocol":"mqv:requestresponse" whereas the old way of doing it with the subscribers and publishers agreeing on a response topic has to be put in the form with something like "mqv:responseTopic": "/mytopic/response"

[JKRhb]

I am currently experimenting with adding WoT functionality to zigbee2mqtt and also came accross some questions regarding read operations.

I think both of the solutions you presented above look very promising for old and new versions of MQTT. Using a response topic is more or less also the way requests are handled in zigbee2mqtt as you can publish to a .../get topic and receive the response over the device's "base" topic (for example zigbee2mqtt/smartlight).

At the moment, the default binding for readproperty operations is specified as SUBSCRIBE. I guess this should/could probably be updated to PUBLISH with the mqv:requestresponse subprotocol you have mentioned? Or do you think there is still some discussion required before integrating it into the binding templates?

[egekorkan]

@JKRhb thank you for the interest in this topic! MQTT did not receive much attention lately by the working group so there was not much discussion on this topic. I had checked the MQTT library of node-wot and it did not have support for this at the time. What zigbee2mqtt is doing is the old way that prompted the MQTT standardization to in the end include a built-in way of doing it. In the end, there is still some discussion to do, and would like to then kickstart it here. I see some points to discuss regarding request-response:

1. MQTT5 implementations: These can and probably should use the built-in mechanism
2. Older implementations: They will define a topic where this can be done
3. Implementations without request-response operations

• Case 1 (MQTT5 implementations): The Thing has to say in its TD that this mechanism is supported, i.e. when the publisher sends to a certain topic with a response topic (which can be anything, thus not in the TD) in the MQTT header, it will respond back. A readable property's MQTT form should thus look like this:

  "forms": [
      {
        "href": "mqtt://example.com",
        "mqv:topic":"myProperty"
        "op": "readproperty",
        "mqv:controlPacketValue": "PUBLISH", //This should probably be omitted since it is implied by the subprotocol
        "subprotocol":"mqv:requestresponse"
      }
    ]

  In this case, when the Consumer published to the topic "myProperty" with a response topic in the header called "myPropertyResponse" (or anything else), the Thing will publish to this topic with the current property value. Of course, the Thing is subscribed to the topic "myProperty" in the first place and the Consumer to the "myPropertyResponse" topic.

• Case 2 (Older implementations): Here it gets a bit tricky since there can be two ways this can be done, not sure what is the most common way.

1. Option 1: The Thing knows in advance where to respond back. This seems to be case with the example in zigbee2mqtt, which is for me a subsubprotocol (with two subs :D). So if this response topic is known in advance and thus hardcoded in the Thing implementation, it can be put in the TD or not (which makes it a real subsubprotocol). Below is an example how this can look like:

   "forms": [
       {
         "href": "mqtt://example.com",
         "mqv:topic":"myProperty"
         "op": "readproperty",
         "mqv:controlPacketValue": "PUBLISH", //This should probably be omitted since it is implied by the subprotocol
         "subprotocol":"mqv:requestresponse",
         "mqv:responseTopic": "myPropertyResponse2" // when this is omitted, we need to indicate this with another subprotocol
       }
     ]

In this case, when the Consumer published to the topic "myProperty", the Thing will publish to the "myPropertyResponse2" topic with the current property value. Of course, the Thing is subscribed to the topic "myProperty" in the first place and the Consumer to the "myPropertyResponse" topic. In addition to the previous case, we also need to indicate that this is not the built-in way. I am not sure what is the best way though. Indicating the version is not enough since I can use MQTT 5 and still do the old stuff.

2. Option 2: The response topic is indicated in the payload. This gets somewhat tricky since we are tempted to put this information in the DataSchema and also make sure that it is not understood as writing to the property, i.e. having an object key with writeOnly set to true would not be good since publishing to read something is not writing to the property. Thus, this DataSchema needs to be in a form. To clarify what I mean, below example is not good:

"oneOf":[{
"type":"string", // this is the response topic
"@type":"mqv:responseTopic",
"writeOnly":true
},{
"type":"string", // this is the usual property value we get
"readOnly":true
}],
"forms": [
        {
          "href": "mqtt://example.com",
          "mqv:topic":"myProperty"
          "op": "readproperty",
          "mqv:controlPacketValue": "PUBLISH", //This should probably be omitted since it is implied by the subprotocol
          "subprotocol":"mqv:requestresponse"
        }
      ]

Thus, we would need something like this:

"forms": [
        {
          "href": "mqtt://example.com",
          "mqv:topic":"myProperty"
          "op": "readproperty",
          "mqv:controlPacketValue": "PUBLISH", //This should probably be omitted since it is implied by the subprotocol
          "subprotocol":"mqv:requestresponse",
          "mqv:payload":{
              "@type":"mqv:responseTopic"
          }
        }
      ]

Not super nice but I think this is quite clear and verbose that it should not be misunderstood. Not sure what would need to be put in the TD if the publish payload contains other data.

• Case 3 ( Implementations without request-response operations): I think we need some discussion here. If an implementation does not have request-response mechanism implemented, what should the property forms look like?

P.S. Sorry for the long comment but I couldn't stop :D If you are further interested in this, we should probably have a discussion in one of the TD/Binding Calls where we can invite you as guest.

[JKRhb]

@egekorkan Thank you for your comprehensive summary of the different cases/options! Looks already very promising! :)

Regarding case 2, maybe there could be a subprotocol with a different name to differentiate between MQTTv5's new request-reponse solution and the legacy case? Or can we let the consumer infer from the missing mqv:responseTopic and mqv:payload fields that in this case MQTTv5 with a response topic in the message is supposed to be used?

Not sure what would need to be put in the TD if the publish payload contains other data.

With regard to case 2, option 2, I was also wondering if indicating the reponse topic in the payload would mean that it is not possible to use any input data in a request-reponse use case (which would probably be mostly relevant in the case of actions that have both an input and an output). If so, I guess these kinds of affordances could only use either case 1 or case 2, option 1?

Also, I guess both options in case 2 should be possible to use to cover a maximum proportion of legacy devices?

Case 3 ( Implementations without request-response operations): I think we need some discussion here. If an implementation does not have request-response mechanism implemented, what should the property forms look like?

Hmm, should a property that does not support request-response only be observable? I guess an alternative would be to subscribe to a topic and then immediately unsubscribe which, however, would only work if the published property is retained at the broker. If this is a valid procedure should it be described by another subprotocol?

P.S. Sorry for the long comment but I couldn't stop :D If you are further interested in this, we should probably have a discussion in one of the TD/Binding Calls where we can invite you as guest.

No worries, it was a very interesting read! :) And I'd be honored to join one of your calls as a guest :)

[egekorkan]

@JKRhb we will invite you to a next call. Discussing today. Also, this request response pattern applies to actions with output as well, i.e. I need to return a payload to an action invocation over MQTT, how can I do it?

[egekorkan]

The consensus in the call of 21.04 was to define the terms for being able to do this. Also, retain should be still considered for reading properties. Since we do not have a view of what are the possible ways and the most popular way to do request-response is, we will not have defaults for readproperty and invokeaction.

I will create a PR with the Case 1 and Case 2 Option 1 above and we can discuss better on this. Some further read to understand what is going on: http://www.steves-internet-guide.com/mqttv5-request-response/ , https://www.hivemq.com/blog/mqtt5-essentials-part9-request-response-pattern/ and list of mqtt clients supporting v5: http://www.eclipse.org/paho/index.php?page=downloads.php

[JKRhb]

@egekorkan Sorry to bother you but have there been any updates on this issue? :) Let me know if I there as any way I could support you with this topic.

[egekorkan]

I have overlooked this somehow and I was on holidays. I will do this in these weeks together with the big restructuring

[JKRhb]

Considering the advances in the MQTT vocabulary, I think we could probably also proceed with this issue? The differentiation between the different cases is probably also relevant with regard to #174.

CC @relu91

[relu91]

Yes, I think this issue is very relevant, considering that I've seen this response pattern in multiple implementations. We should support it.