Is the use of "$ref" and "definitions" allowed in TD ?

ToruKawaguchi commented 5 years ago

Can anybody clarify whether it is possible to use "$ref" and "definitions" of JSON schema within a TD or not? This is partly mentioned in https://github.com/w3c/wot-thing-description/issues/151#issuecomment-407753552, but was not clear to me what the conclusion is, so I am raising a new issue.

Some existing REST based API requires not only property value itself but also some predefined fields in request/response body, such as id and type of the thing. It is following a design practice to make developers easy to understand what that data is for. In addition, HATEOAS approach requires links field in every response body.

Considering such use case, it is helpful if we could define a data schema separately from each property definition, from both TD readability and size point of view.

In addition, above use case is not suitable for automatic construction of data field by "readall" discussed in #151, so if we want to read all property at once we need to manually define a special property which includes all other properties. In such case, referencing the definition out from that property could reduce the size of TD.

Below is an experimental TD applying $ref and definitions according to above use case just for discussion:

{
    "id": "urn:dev:wot:com:example:servient:homeAirConditioner:1",
    "name": "MyHomeAirConditioner",
    "security": [
        {
            "scheme": "basic"
        }
    ],
    "base": "https://www.example.com/devices/homeAirConditioner/1/",
    "properties": {
        "": {
            "title": "All properties",
            "description": "Retrieves or sets all properties at once",
            "type": "object",
            "properties": {
                "_id": {
                    "$ref": "#/definitions/_id"
                },
                "_type": {
                    "$ref": "#/definitions/_type"
                },
                "_properties": {
                    "type": "object", 
                    "properties": {
                        "operationStatus": {
                            "$ref": "#/definitions/operationStatus"
                        },
                        "targetTemplatureCelcius": {
                            "$ref": "#/definitions/targetTemplatureCelsius"
                        }
                    }
                },
                "links": {
                    "$ref": "#/definitions/links"
                }
            },
            "forms": [
                {
                    "href": "properties"
                }
            ]
        },
        "operationStatus": {
            "type": "object",
            "properties": {
                "_id": {
                    "$ref": "#/definitions/_id"
                },
                "_type": {
                    "$ref": "#/definitions/_type"
                },
                "_properties": {
                    "type": "object", 
                    "properties": {
                        "operationStatus": {
                            "$ref": "#/definitions/operationStatus"
                        }
                    }
                },
                "links": {
                    "$ref": "#/definitions/links"
                }
            },
            "forms": [
                {
                    "href": "properties/operationStatus"
                }
            ]
        },
        "targetTemplatureCelsius": {
            "type": "object",
            "properties": {
                "_id": {
                    "$ref": "#/definitions/_id"
                },
                "_type": {
                    "$ref": "#/definitions/_type"
                },
                "_properties": {
                    "type": "object", 
                    "properties": {
                        "targetTemplatureCelsius": {
                            "$ref": "#/definitions/targetTemplatureCelsius"
                        }
                    }
                },
                "links": {
                    "$ref": "#/definitions/links"
                }
            },
            "forms": [
                {
                    "href": "properties/targetTemplatureCelsius"
                }
            ]
        }
    },
    "definitions": {
        "_id": {
            "title": "ID",
            "description": "ID of this device",
            "type": "string",
            "const": "urn:dev:wot:com:example:servient:homeAirConditioner:1"
        },
        "_type": {
            "title": "Type",
            "description": "Type of this device",
            "type": "string",
            "const": "homeAirConditioner"
        },
        "operationStatus": {
            "title": "Operation Status",
            "description": "Operational Status such as On and Off",
            "type": "boolean"
        },
        "targetTemplatureCelsius": {
            "title": "Target Temperature Celsius",
            "description": "Target Temperature in Celsius",
            "type": "number",
            "minimum": 0.0,
            "maximum": 50.0,
            "unit": "Celsius"
        },
        "links": {
            "title": "Links",
            "description": "Possible links of this device",
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "rel":  {
                        "type": "string"
                    }, 
                    "href": {
                        "type": "string" 
                    }
                }
            }, 
            "readOnly": true
        }
    }
}

egekorkan commented 5 years ago

I had a similar problem before and this was sadly not the solution. This approach would require special parsing method. In other words, a JSON Schema parser/validator will not recognize the definitions key since it is outside the "JSON Schema" found in the properties. As you said, it is experimental but I am also interested on possible solutions to this problem.

handrews commented 5 years ago

This reminds me that I want to put something in JSON Schema to facilitate embedding JSON Schema in other documents. It won't get done in this year's last draft (we're almost out of year), but should get some attention in the early 2019 draft.

mmccool commented 5 years ago

We’re almost out of time this cycle as well for the TD. So maybe we should incubate this for the next round, which means intercepting updates to JSON Schema is a possibility.

It would be a good idea to understand what kinds of REST APIs a TD can’t model without this and whether they are common in practice in IoT contexts.

ToruKawaguchi commented 5 years ago

@egekorkan @handrews @mmccool Thank you for your comments ! I understood that above approach is currently beyond JSON-Schema and consequently beyond Thing Description specification. For now, I would repeat schema definitions in each property.

sebastiankb commented 5 years ago

I'm agree. This seems to be an issue that cannot be solved quickly. I will mark it for a candidate for the next TD version.

vcharpenay commented 4 years ago

For the record: the JSON-LD mechansim of addressing objects (with @id) is (mostly, if not entirely) equivalent to that of JSON schema. Only the syntax differs. In future TD specs, compatibility with both should be retained as much as possible.

In the docs of the RDF vocabulary for JSON schema, there is an example of schema reference done with @id: https://www.w3.org/2019/wot/json-schema#referencing-and-linking.

takuki commented 4 years ago

In the docs of the RDF vocabulary for JSON schema, there is an example of schema reference done with @id: https://www.w3.org/2019/wot/json-schema#referencing-and-linking.

@vcharpenay , if this is a practice already used by OCF, can't we safely assume it does work?

vcharpenay commented 4 years ago

@vcharpenay , if this is a practice already used by OCF, can't we safely assume it does work?

You mean that $id/$ref is used by OCF, so it should be supported by the TD? To some extent, JSON Schema's $-notation is embeddable in JSON-LD, so that shouldn't be a problem to introduce it in a future TD spec.

takuki commented 4 years ago

You mean that $id/$ref is used by OCF, so it should be supported by the TD?

I was looking at Example 4 in RDF vocabulary for JSON schema document. The example appears to be using something different from $id/$ref ,

sebastiankb commented 4 years ago

we should exchange also with WISHI since they have also some discussion about $ref and definitions for SDF

sebastiankb commented 4 years ago

@sebastiankb @danielpeintner will provide a proposal how this can introduce in TDs. It will also include requirements and use cases for the need. The proposal will be also syncronized with @ToruKawaguchi

danielpeintner commented 4 years ago

@sebastiankb and I have been tinkering around and would like to share our findings.

Since a TD is compliant with the definition of JSON schema document the validation of properties works out-of-the box. Hence we suggest to introduce the new term "definitions" on the top level just like JSON schema does.

Let's use the TD example below. You can feed it to any JSON schema validator and it will check whether the properties billingAddress and shippingAddress are compliant to the definition of address.

{
    "@context": "https://www.w3.org/2019/wot/td/v1",
    "id": "urn:dev:ops:32473-MyCompany-1234",
    "title": "MyCompany",
    "securityDefinitions": {"basic_sc": {
        "scheme": "basic",
        "in": "header"
    }},
    "security": ["basic_sc"],
    "definitions": {"address": {
        "type": "object",
        "properties": {
            "streetAddress": {"type": "string"},
            "city": {"type": "string"},
            "state": {"type": "string"}
        },
        "required": [
            "streetAddress",
            "city",
            "state"
        ]
    }},
    "properties": {
        "billingAddress": {
            "$ref": "#/definitions/address",
            "forms": [{"href": "https://mycompany.example.com/properties/billingAddress"}]
        },
        "shippingAddress": {
            "$ref": "#/definitions/address",
            "forms": [{"href": "https://mycompany.example.com/properties/shippingAddress"}]
        }
    },
    "actions": {"setBilling": {
        "input": {"$ref": "#/definitions/address"},
        "output": {"$ref": "#/definitions/address"},
        "forms": [{"href": "https://mycompany.example.com/actions/setBilling"}]
    }},
    "events": {"billingAddressChange": {
        "data": {"$ref": "#/definitions/address"},
        "forms": [{"href": "https://mycompany.example.com/events/billingAddressChange"}]
    }}
}

Hence one can copy it for example to https://jsonschemalint.com/ and the following instance is properly validated.

{
  "billingAddress": {
    "streetAddress": "Otto-Hahn-Ring",
    "city": "Munich",
    "state": "GER"
  }
}

So far so good. What is not checked is the input / output of actions and DataSchema terms for events like data.

To achieve proper checking for action/event DataSchema snippets the proposed approach is to copy the entire "definitions" snippet next to "$ref". Doing so once again any JSON schema validator can check the validation.

Does this overhead of copying (if someone wants to prove validation) seem acceptable?

Any other thoughts or improvements we could make. We are especially interested in ideas from @handrews since he mots likely has the most expertise in this area.

sebastiankb commented 4 years ago

based on the discussion during the vf2f meeting we should careful with the $ref usage such as circles. We may to think about to constrain the usage of that kind of features (especially for constrained devices).

sebastiankb commented 4 years ago

in vf2f meeting it was also requested to collecting more use cases and should be documented

sebastiankb commented 3 years ago

From today's TD call:

to avoid backwards compatible problems we should defer this to 2.0
however, we can introduce this feature in the Thing Model
we should also evaluate if we should follow the sdf approach since this is not limit to JSON Schema definitions
we should also check the new approach is working with JSON-LD
@danielpeintner @vcharpenay please can you have a look on that

handrews commented 3 years ago

Note that definitions has been changed to $defs in the last two drafts (current draft: https://tools.ietf.org/html/draft-bhutton-json-schema-00 ). The core vocabulary all uses $ prefixes, and since we were renaming anyway I decided I was tired of accidentally typing "definitoins" :-)

danielpeintner commented 3 years ago

Note that definitions has been changed to $defs in the last two drafts

Thank you for your feedback. I wonder whether this change is just syntactic sugar meaning that even in the past someone could have used $defs instead of definitions. This is at least my understanding. Users are free to choose where the referenced definitions are located. Isn't this the case?

handrews commented 3 years ago

@danielpeintner Yes, that's essentially true. The differences is that $defs cannot be redefined by an extension vocabulary, while definitions could be. 2019-09 and 2020-12 define keywords (including most of the "standard" keywords) as vocabularies, which are declared and described in meta-schemas. The only vocabulary that MUST always be supported no matter what meta-schema you use is the core vocabulary: $schema, $vocabulary, $id, $anchor, $ref, $dynamicAnchor, $dynamicRef, $defs, and $comment. These are the keywords needed to bootstrap JSON Schema processing, and (in the case of $defs and $comment) reserve safe locations for schemas and non-end-user-visible comments that cannot be redefined to do something else.

The formal definition of $defs also ensures that the schemas underneath it are recognized to be schema objects, including when nested in subschemas. This is needed in order to properly handle the presence of $schema, $id, $anchor, or $dynamicAnchor when determining what valid reference target URIs exist:

Here's an example using YAML for brevity, with the standard meta-schema plus a keyword someUndeclaredKeyword which is not specified in any vocabulary (this is still legal, but as you will see it is not a great idea):

$schema: "https://json-schema.org/draft/2020-12/schema"
$id: "https://example.com/foo"
type: object
properties:
  foo:
    $ref: "https://example.com/bar"
  oof:
    $ref: "https://example.com/rab"
$defs:
  bar:
    $id: "https://example.com/bar"
    $comment: "pretend some cool schema keywords are here"
someUndeclaredKeyword:
  rab:
    $id: "https://example.com/rab"

In this case, when the implementation loads the schema, it will know that things under $defs are schemas, so it will process the $id under bar, and be able to resolve the reference for foo. However, unrecognized keywords are treated as annotations (information to be associated with the instance, like title), so the implementation won't realize that it should treat the $id under rab as a schema identifier. And therefore it won't be able to resolve the reference in oof.

This is rather advanced usage so in practice it's not a problem for many people. However, this $id usage pattern does appear when bundling multiple schema resources into a single schema document.

You could, of course, define an extension vocabulary that specifies the semantics of definitions to be identical to $defs, and then implementations that know that extension would handle either keyword correctly. But the implementation would need to be aware of that vocabulary.

A lot of what went into 2019-09 and 2020-12 with the keyword taxonomy (identifiers, references, assertions, annotations, applicators, and reserved locations) ensure that keyword behavior is sufficiently well-defined in general that folks can successfully design extensions that fit with the overall system, so things like "which parts of this document are a schema and how does the implementation know that" have been a focus.

sebastiankb commented 3 years ago

The $ref-based feature is introduced for Thing Models. I propose to evaluate the ref concept for TD in the TD 2.0 version again.

w3c / wot-thing-description

Is the use of "$ref" and "definitions" allowed in TD ? #307