Constraints for the use of `cov:observe`

[JKRhb]

Dealing with CoAP forms for observable properties and events that were missing the cov:observe protocol in https://github.com/eclipse-thingweb/node-wot/pull/1214, I wondered if the Binding Template document should specify that the use of the cov:observe subprotocol is only permitted for certain operation types (i.e., observeproperty, unobserveproperty, subscribeevent, and unsubscribeevent) or if it should also be allowed in forms that contain readproperty and/or writeproperty.

I suppose at least readproperty could be allowed together with cov:observe since in the case of CoAP, the observeproperty operation also implies a readproperty operation. I compiled a list of examples below based on which we could discuss this in more detail.

{
  ...
  "properties": {
    "foo": {
      "description": "Should any of the following forms be allowed?",
      "observable": true,
      "forms": [
        {
          // Could be valid, but should it be?
          "href": "/foo",
          "op": ["readproperty", "writeproperty", "observeproperty"],
          "subprotocol": "cov:observe"
        },
        {
          // Could be valid
          "href": "/foo",
          "op": ["readproperty", "observeproperty"],
          "subprotocol": "cov:observe"
        },
        {
          // This one should probably not be valid
          "href": "/foo",
          "op": ["writeproperty"],
          "subprotocol": "cov:observe"
        },
        {
          // Could be valid
          "href": "/foo",
          "op": ["readproperty"], 
          "subprotocol": "cov:observe"
        },
      ]
    }
  },
  ...
}

[egekorkan]

Good points. Some related issues:

• HTTP Default event: https://github.com/w3c/wot-binding-templates/issues/199
• MQTT retain: https://github.com/w3c/wot-binding-templates/issues/83

Regarding the forms above:

• Form 3 should not be allowed as it is not possible to write with observation. Same with form 1 with a possible exception (see below)
• Form 4 should be recommended against. It is possible but misleading.
• Form 2 is fine but I would also prefer to see it where we split the form to two, one per operation where readproperty doesn't have subprotocol.

The overall issue is that we do not say if subprotocol applies to all operations below or not. I think we should add that assertion. If we don't, form 1 can be possible.

At the same time, I am realizing that we need a new term that indicates whether observation will return the current value before sending observations or not. This would be the same for mqtt. A very simple example:

{
  "href": "/foo",
  "op": ["observeproperty"],
  "subprotocol": "cov:observe",
  "currentValueEmission" : true
}

`currentValueEmission` implies that the current value will be sent. Intentionally not prefixed with a protocol ontology, thus a core TD term

[ektrah]

As always, I don't think it's possible to generically tell a TD consumer in a TD how CoAP works; a consumer must have a basic understanding of what it wants and how CoAP works.

• If a TD consumer wants to read a property and the CoAP server provides the value of the property in response to a GET request, then the consumer should use a GET request.

  If the TD consumer wants to read a property more often, it can also use a GET request with an Observe option to always have a response in the cache.

  On the other hand, if the server provides the property value as a response to a POST request, then the Observe option cannot be used as per the protocol definition.

• If a TD consumer wants to observe a property and the CoAP server provides the value of the property in response to a GET request, then the consumer should use a GET request with the Observe option. As long as the server supports the option and has capacity for the client, the client receives notifications; otherwise the client must poll at regular intervals to satisfy its need.

  On the other hand, if the server provides the property value as a response to a POST request, then the Observe option cannot be used as per the protocol definition.

• If a TD consumer wants to write a property and the CoAP server accepts the value of the property as a POST request, then the consumer should use a POST request.

  On the other hand, if an implementer somehow manages to get a CoAP server to accept the value of the property as a GET request with an Observe option (don't ask me how), then it should be possible to express that in a TD as well.

In summary, I don't think you have to specify normatively that the use of the cov:observe subprotocol is only permitted for certain operation types, because that just follows from the protocol definition. However, it might be useful to describe informatively where the use of the cov:observe subprotocol makes most sense.

[JKRhb]

Thank you for the summary, @ektrah! As mentioned by @egekorkan above, the core issue probably relates to the question of how a subprotocol should be handled – so in this case if the consumer encounters a property with a writeproperty operation and a cov:observe subprotocol, should it just send the POST request and ignore the subprotocol (because it knows that this does not make sense here)? Or should it treat the form as invalid, and then either throw an error or skip it if another form exists?

Adding some informative text on the use of cov:observe might actually be sufficient for the CoAP Binding Template, while the subprotocol handling, in general, should probably be dealt with in the TD specification and/or the Scripting API specification.

[ektrah]

Since this issue is specifically about CoAP: Maybe we should think about removing the cov:observe subprotocol altogether.

• A TD producer cannot know whether a TD consumer wants to read the value of the property only once or more often. So the subprotocol can only specify whether the server understands the Observe option in principle, but not whether the client should set the Observe option in its request.
• Regardless of whether the server understands the observe Option in principle or currently has capacity for a client, a TD consumer that needs a current property value several times or continuously over a longer period of time should just poll the resource. If the cov:method specified for the operation is GET, the CoAP client should always set the Observe option, minimizing the network traffic by pausing its requests while it keeps receiving notifications.

[JKRhb]

I think removing cov:observe as a subprotocol sounds good. Maybe we could introduce a vocabulary term like supportsObserve instead? This way, it would be a bit clearer that it is up to the consumer to decide whether they want to use polling or rely on the Observe mechanism for updates instead.

[ektrah]

If the client has decided that it wants to perform the observeproperty operation and sees that this is done using the CoAP GET method, I'm not sure it needs any further decision-making help:

1. If the server supports the Observe option and has capacity for the client, the client should use the Observe option. If the server subsequently stops sending notifications, the client should poll with the Observe option set until it receives notifications again.
2. If the server supports the Observe option and does NOT have capacity for the client, the client should still use the Observe option and poll with the Observe option set until the server has capacity.
3. If the server does NOT support the Observe option, the client has to poll. There is no harm in using the Observe option here, as it will simply be silently ignored.

So using the Observe option works in all three cases. In pseudo-code:

if (operation == observeproperty && cov:method == GET) {
    do {
        var request = new CoapRequest(GET, href);
        request.setObserveOption(0);
        request.send();
        do {
            var response = await request.nextResponse();
            if (response.code != 2.05) {
                throw new CoapException();
            }
            yield response.payload;
        }
        while (observing && response.hasObserveOption);
    }
    while (observing);
}

(Congestion control and rate-limiting omitted for brevity.)

[JKRhb]

Thanks, @ektrah, your comment is a very good clarification, and could also be worth adding to the Binding Template document itself.

One aspect that just came to my mind was the handling of events. Could we simply apply the same mechanism here without adjustments (i.e., treat it as a normal observe/polling operation)? Or should we maybe explicitly mention that, especially in the case of events, a server might choose to only acknowledge a confirmable request first and send the event notification response as soon as a new event actually occurs? In that case a consumer might need to wait for a longer period of time after the initial acknowledgement until the notification arrives.

[ektrah]

In CoAP, there are notifications for only one type of event: the change in the state of the observed resource. (And these notifications are not lossless, but rather on a "best effort" basis, with eventual consistency.)

If an application wants to notify about any other type of event (or needs a lossless history of all events that occurred), then CoAP provides a building block that can be used, but not a complete solution.

A possible solution is to append the application-level events to a log and observe this log as a CoAP resource. This can be done relatively efficiently with pagination, i.e. if you divide the log into several resources like a linked list and only observe, for example, a resource that contains a pointer to the newest log entry (which contains a pointer to the previous log entry, and so on).

So, yes, the mechanism for events is exactly the same as for polling or observing a property, except that after the client detects a state change (the observed resource points to a new end of the linked list) it may also have to retrieve the representation of the event(s) that occurred since it detected a state change the last time.

[egekorkan]

Sorry that I could catch up only now. In essence, we want to avoid guesswork on the Consumer side and guide producers to create correct TDs. Some comments to points made above:

@ektrah wrote:

1. If the server supports the Observe option and has capacity for the client, the client should use the Observe option. If the server subsequently stops sending notifications, the client should poll with the Observe option set until it receives notifications again.
2. If the server supports the Observe option and does NOT have capacity for the client, the client should still use the Observe option and poll with the Observe option set until the server has capacity.

Since the TD is mostly static and the instruction to the Consumer is the same, sort of. In general, we should add the guidance for the Consumer to poll when the observation doesn't seem to work due load (case 2) or random reason (case 1).

So the subprotocol can only specify whether the server understands the Observe option in principle, but not whether the client should set the Observe option in its request.

These are not exactly true. The consumer should set Observe option to observe a property. Otherwise it is acting against the TD, breaking the contract.

If the TD consumer wants to read a property more often, it can also use a GET request with an Observe option to always have a response in the cache.

Even though a Consumer is able to do, it should not. It would be breaking the contract. TD is not a guidance but more rules.

There is no harm in using the Observe option here, as it will simply be silently ignored.

Consumer should not set Observe option even though it is not going to be technically fine.

All the three above are assertions in the TD spec (Consumer should build correct requests etc.). I would be as specific as possible for the Consumer implementations which we can control.

[ektrah]

@egekorkan, CoAP was intentionally designed so that when a CoAP client wants to observe a CoAP resource, setting the Observe option in a GET request is always a correct request, without the client having to know in advance whether the CoAP server supports the option in general, has capacity at the moment, or neither. If the correct use of CoAP is not allowed by the TD spec, then we should remove CoAP support from WoT entirely.

[egekorkan]

What I mean is that flexibility and not having to know everything in advance is good but with TDs we have a way to know quite a bit of stuff in advance and can avoid some requests or request configurations as much as possible. A coap binding implementation in WoT should not try to observe a resource if there is no observeproperty operation and/or if the observable is set to false. Doing so would be wrong WoT application in the first place.

If the correct use of CoAP is not allowed by the TD spec, then we should remove CoAP support from WoT entirely.

I wouldn't be so strict on this :) We can mandate/recommend how Consumer applications should behave but cannot do it for Things. Is there a reason why we should say "A CoAP-supporting Consumer can still to try observe even if the observable is set to false"?

[JKRhb]

So, yes, the mechanism for events is exactly the same as for polling or observing a property, except that after the client detects a state change (the observed resource points to a new end of the linked list) it may also have to retrieve the representation of the event(s) that occurred since it detected a state change the last time.

Thanks again for the summary and clarification. This confirms a feeling I had that these kinds of application-level events that are more than mere state changes might not be that well describable using a TD in this context. Using the current TD vocabulary, it seems to me that event affordances do not really make a difference compared to observable properties when using CoAP. I wonder if we could reuse or define a standardized mechanism (similar to the one you described) here to be able to also describe more complex event affordances.

[ektrah]

@egekorkan: If you assume that a TD consumer will mindlessly copy the form fields into a message and send it out without understanding what it's doing, then yes, we shouldn't say that.

But if a TD consumer knows that it will be performing multiple readproperty operations in the near future and the CoAP library implements an optimization that is explicitly mentioned in the protocol spec (namely observing a resource in the background just to keep a cache up to date), then I'm not sure why we should say that is not allowed to use this library.

Anyway. The previous discussion was more about the question of how a TD consumer needs to know in advance in the case of CoAP, i.e. how much information needs to be included in the TD. And I think that it is quite sufficient if a TD consumer knows that it wants to perform the readproperty or observeproperty operation, and the TD gives the consumer the information to do this using the CoAP GET method. The information "subprotocol": "cov:observe" or supportsObserve in the TD provides no added value here in my opinion.

Of course, we can discuss whether we might want to offer a way in TDs to instruct a TD consumer to NOT use certain CoAP features, even if both CoAP client and CoAP server support the feature (for example, because the TD producer produces different TDs for different consumers and by policy wants them to do different things). In the case of the observeproperty operation, though, it would not be clear to me why a TD consumer should be instructed to never set the Observe option. A GET request with Observe Option can always fall back to a normal GET request if the server is unable or unwilling to act on the option, so there is nothing you can preempt here to avoid unnecessary requests or request configurations.

What do you think?

[ektrah]

@JKRhb: Have a look at draft-bormann-t2trg-stp and draft-ietf-core-coap-pubsub. Figuring out how to describe these in a TD would be an interesting exercise 😃

[egekorkan]

But if a TD consumer knows that it will be performing multiple readproperty operations in the near future and the CoAP library implements an optimization that is explicitly mentioned in the protocol spec (namely observing a resource in the background just to keep a cache up to date), then I'm not sure why we should say that is not allowed to use this library.

That would be fine from my perspective. It would be cool to see this kind of things happening actually ;)

The information "subprotocol": "cov:observe" or supportsObserve in the TD provides no added value here in my opinion.

If there is only way to do observation, I agree. It would be like a default value that no one needs to use until there is another way to observe. However, wouldn't it make sense to have coap multicast in some cases or the pubsub in the link above? In those cases, we would need to instruct what the Consumer should do.