how to handle multiple forms

[mlagally]

Do we need to define a set of rules how multiple forms are to be handled by the consumer?

IMHO we need a defined algorithm to ensure interop.

Please see comment in: https://github.com/w3c/wot-profile/pull/125#issuecomment-966589941

[sebastiankb]

Just for information, why multiple forms entries sometimes are needed:

1. properties may have different resources for read, write, observe and unobserve.
2. A Thing can have a local IP (reachable only in a local network) and if intended, a global one (accessible from everywhere in the Internet).
3. supports IPv4 and IPv6

[danielpeintner]

FYI: The scripting API defines the following steps E.g., for readProperty(...) see Step 5 in https://w3c.github.io/wot-scripting-api/#the-readproperty-method

• If option's formIndex is defined, let form be the Form associated with formIndex in interaction's forms array
  • Note: we allow a so called formIndex option in case a specific form should be taken
• otherwise let form be a Form in interaction's forms whose op is readproperty, selected by the implementation.
  • the implementation can chose the right match (supported protocol) and op readproperty in this case

Note: node-wot goes in order and picks the first that matches.

[benfrancis]

@mlagally wrote:

Do we need to define a set of rules how multiple forms are to be handled by the consumer. IMHO we need a defined algorithm to ensure interop.

I agree this is needed, but ideally it would be provided by the Thing Description specification to solve the general problem of how a Consumer should choose between multiple forms provided for the same interaction affordance and/or operation. I have proposed this as a deliverable for the next charter period in https://github.com/w3c/wot/issues/978#issuecomment-965127375

In the meantime we could address this in the Core Profile protocol binding. There is already text under each operation which provides criteria for selecting a Form for a given operation, but it doesn't currently say what to do if multiple forms match those criteria. E.g. for the readproperty operation it says:

The URL of a Property resource to be used when reading the value of a property MUST be obtained from a Thing Description by locating a Form inside the corresponding PropertyAffordance for which:

1. After defaults have been applied, its op member contains the value readproperty
2. After being resolved against a base URL where applicable, the URI scheme [RFC3986] of the value of its href member is http or https

The resolved value of the href member MUST then be used as the URL of the Property resource.

I could work on extending those assertions to say what to do if there are multiple matches, but first we need to agree on what should happen!

Please see comment in: #125 (comment)

This comment is about a proposed way forward for resolving the data model debate. I provided feedback on the multiple forms issue in https://github.com/w3c/wot-profile/pull/131#pullrequestreview-807153424, perhaps you meant to link to that.

@sebastiankb wrote:

Just for information, why multiple forms entries sometimes are needed:

1. properties may have different resources for read, write, observe and unobserve.
2. A Thing can have a local IP (reachable only in a local network) and if intended, a global one (accessible from everywhere in the Internet).
3. supports IPv4 and IPv6

4. A Thing should be able to support multiple profiles with different protocol bindings (e.g. Core Profile using HTTP + Constrained Profile using CoAP+CBOR)

@danielpeintner wrote:

FYI: The scripting API defines the following steps E.g., for readProperty(...) see Step 5 in https://w3c.github.io/wot-scripting-api/#the-readproperty-method

• If option's formIndex is defined, let form be the Form associated with formIndex in interaction's forms array
  • Note: we allow a so called formIndex option in case a specific form should be taken
• otherwise let form be a Form in interaction's forms whose op is readproperty, selected by the implementation.
  • the implementation can chose the right match (supported protocol) and op readproperty in this case

OK, so this is roughly equivalent to what the current Core Profile protocol binding says too. Pick the form with a readpropery operation which uses the HTTP protocol. It doesn't define what to do in the case that multiple forms match those criteria.

Note: node-wot goes in order and picks the first that matches.

Hah, the front end of WebThings Gateway currently picks the last one, for internal implementation reasons (because add-on adapters can add their own forms and we know the last one is the one that the gateway added itself).

What do other implementations do?

CC @relu91

[danielpeintner]

I don't believe specifying every granularity helps to ensure interoperability.

If 2 (or more) forms fulfill the requirements (e.g., right protocol, op value etc) I don't really see why there should be a need to specify which form is to be taken. If the form matters there is a more fundamental problem... don't you think?

[benfrancis]

@danielpeintner I'm sorry I don't think I understood your comment.

Are you saying that you don't think it matters which form a Consumer picks if there are multiple matches (i.e. it's OK if two different Consumers looking for the same combination of operation and protocol would behave differently for the same Thing Description)?

Or are you saying that a Thing shouldn't be providing those redundant forms in the first place? If so, that still doesn't explain what a Consumer should do if that does happen.

[danielpeintner]

Are you saying that you don't think it matters which form a Consumer picks if there are multiple matches (i.e. it's OK if two different Consumers looking for the same combination of operation and protocol would behave differently for the same Thing Description)?

Yes, that's what I mean.

Let me make an analogy.

There are situations where the form matters, e.g, the order in the case of an mathematical subtraction. It is important that minuend comes before the subtrahend (minuend - subtrahend = difference). This is the case of picking the right protocol, op et cetera.

There are other situation where the form should not matter, e.g., in the case of an mathematical addition. The order of summands is not of importance (summand1 + summand2 == summand2 + summand2 == sum) This (at least for me) is the case for forms that provide the same protocol/op.

IF in this case the order still matters a "requirement" is missing that makes them behave differently...

[egekorkan]

Just an example would be the forms having everything the same but one is using IPv4 and other IPv6

[mlagally]

@egekorkan @danielpeintner The handling of multiple forms is still very unclear to me. To give an example: If you have a list of 10 forms and agree to process them sequentially until you find a form that works, you may end up in trying 9 forms that don't connect within a timeout. Using typical timeouts of 10-15s this means that you may wait for >100s until you get a response. How is that supposed to work the 2nd time? Reiterate the same list again, or use the form that worked last time? In which order? Start from the beginning again?

[egekorkan]

I think that node-wot can internally implement something that saves the working form and use it next time but it is not implemented now at least. Also, I don't think that it goes to the next form if the current one fails due to a timeout, it shows an error. It simply tries to find a form whose protocol it supports and goes for it.

I wrote this with the knowledge from the top of my head but @relu91 and @danielpeintner can confirm or show the actual way it happens.

Regarding what @danielpeintner wrote above:

If 2 (or more) forms fulfill the requirements (e.g., right protocol, op value etc) I don't really see why there should be a need to specify which form is to be taken. If the form matters there is a more fundamental problem... don't you think?

Sort of true that the underlying problem should be fundamental. Just with IPv4 vs IPv6, it is rather funny that sometimes network providers (especially mobile networks) support only one or both. This means that the address resolution may not work for some clients and do for others. For the devices we host in our lab, I can overengineer and put 4 forms:

1. Public IPv4 address: When DNS resolution fails and client must use ipv4
2. Public IPv6 address: When DNS resolution fails and client must use ipv6
3. Public domain name: When DNS resolution works
4. Local IPv4 address: Any public IP or domain routing fails due to some network configuration at TUM.

This is definitely over engineered especially when public domain name is also available but if we could not get a domain name and wanted to address all use cases, we would have 3 forms with href having options 1, 2 and 4. In practice, we do not overengineer anything go with domain name in one TD and local IP in another TD.

[danielpeintner]

I think that node-wot can internally implement something that saves the working form and use it next time but it is not implemented now at least. Also, I don't think that it goes to the next form if the current one fails due to a timeout, it shows an error. It simply tries to find a form whose protocol it supports and goes for it.

I wrote this with the knowledge from the top of my head but @relu91 and @danielpeintner can confirm or show the actual way it happens.

Your recollection is true. It tries to find the right form (other forms are discarded). Once there is a form that matches the requirements it uses it and in case of failure it reports the error. No other form is tried.

[benfrancis]

@danielpeintner wrote:

Are you saying that you don't think it matters which form a Consumer picks if there are multiple matches

Yes, that's what I mean.

Ah OK. My instinct had been that this is something that we'd want to be consistent between Consumers, but maybe it's OK for Consumers to define their own set of heuristics for picking Forms based on the protocols and data formats they support, and other factors like resource constraints. There are certainly other parts of the web which behave like that, with different user agents expressing different preferences of data formats in HTTP Accept headers, or different installability criteria for web app installation for example. These are intentionally left as areas in which different user agents can diverge and compete.

So as far as the Core Profile is concerned, perhaps the existing criteria for selecting a form are enough and beyond that it's down to the Consumer to choose.

Regarding the specific case of providing both local and remote hosts for the same endpoints, what WebThings does in that case is to provide two separate Thing Descriptions. There's the local version using a .local domain and the remote version using a *.webthings.io domain and as far as a Consumer is concerned they are two separate Web Things. When the gateway directory is accessed via the local domain it lists the local versions and when accessed via the remote domain it lists the remote versions.

@mlagally wrote:

If you have a list of 10 forms and agree to process them sequentially until you find a form that works, you may end up in trying 9 forms that don't connect within a timeout. Using typical timeouts of 10-15s this means that you may wait for >100s until you get a response. How is that supposed to work the 2nd time? Reiterate the same list again, or use the form that worked last time? In which order? Start from the beginning again?

I don't think anyone is suggesting that a Consumer should try all the forms in sequence until one works, just that they may need to pick from multiple forms providing the same operations using different protocols and data formats. E.g. Pick between HTTP & CoAP or HTTP & WebSockets or JSON & XML or JPEG and WebP or Ogg Vorbis & MP3.

In the Core Profile we already specify that Consumers should pick a form which uses HTTP. The default content format of JSON is implied, but we could make that more explicit in case a Consumer offers the same operations with XML as well for example.

We don't currently say anything about preferred binary formats for things like images, audio and video but I'm not sure we want to mandate anything there since it may unnecessarily complicate Consumer conformance and trying to pick preferred formats is a bit of a minefield.

[mlagally]

In the Core Profile we already specify that Consumers should pick a form which uses HTTP. The default content format of JSON is implied, but we could make that more explicit in case a Consumer offers the same operations with XML as well for example.

I agree with this proposal, that can be easily implemented.