Consider using SWE Common to describe Datastream result structure

[alexrobin]

The current Datastream model only allows for scalar results (a single observed property) and creating a separate collection for the MultiDatastream case feels a little hacky. Can we consider using SWE Common to provide the result structure like what was done in the Tasking extension?

This would allow unifying Datastream and MultiDatastream collections and cover use cases with simple scalar results, but also vector results (e.g. position, orientation, any vector quantity such as stress or strain, etc.) and array results (e.g. images, coverages, observation profiles, etc.). Note that with the alignment to O&M v3, the collection may just be named "ObservationCollections" instead of "Datastreams".

To keep the core simple, more complex result structures can be mandated by separate conformance classes (e.g. core only mandates support for scalar results, additional conformance classes defined for records of scalars / vectors, nested records, array results, etc.).

Below are examples of how different cases of Datastreams could look like in JSON:

Example with scalar result

{
  "name": "Sensor 201 - Air Temperature",
  "description": "Datastream of temperature measurements",
  "resultType": {
    "type": "Quantity",
    "definition": "/v1.0/ObservedProperties(155)",
    "label": "Air Temperature",
    "uom": {
        "code": "Cel",
        "symbol": "°C",             // not in SWE Common but could be added in v2.1
        "label": "Degree Celsius",  // not in SWE Common but could be added in v2.1
        "definition": "http://www.qudt.org/qudt/owl/1.0.0/unit/Instances.html#DegreeCelsius"
    },
    "ObservedProperty@iot.navigationLink": "/v1.0/ObservedProperties(155)" // redundant with definition property
  },
  "Sensor@iot.navigationLink": "/v1.0/Sensors(201)",
  "Thing@iot.navigationLink": "/v1.0/Things(1)"
}

Note that the definition property of SWE Common can be used both to refer to an observed property defined in an external ontology or to one defined locally in SensorThings. We can keep the ObservedProperty navigation link in addition if necessary.

A corresponding Observation would look like:

{
  "@iot.id": "f4f87548eqqz",
  "phenomenonTime" : "2021-04-15T06:14:00.829Z",
  "resultTime" : "2021-04-15T06:14:00.829Z",
  "result" : 14.8,
  ...
}

Example with vector result

{
  "name": "Sensor 56 - Platform Attitude",
  "description": "Datastream of platform orientation measurements",
  "resultType": {
    "type": "Vector",
    "definition": "http://www.opengis.net/def/property/OGC/0/PlatformOrientation",
    "referenceFrame": "http://www.opengis.net/def/cs/OGC/0/NED",
    "coordinates": [
      {
        "name": "heading",
        "type": "Quantity",
        "definition": "http://sensorml.com/ont/swe/property/TrueHeading",
        "axisID": "Z",
        "label": "Heading Angle",
        "uom": {
          "code": "deg"
        }
      },
      {
        "name": "pitch",
        "type": "Quantity",
        "definition": "http://sensorml.com/ont/swe/property/PitchAngle",
        "axisID": "Y",
        "label": "Pitch Angle",
        "uom": {
          "code": "deg"
        }
      },
      {
        "name": "roll",
        "type": "Quantity",
        "definition": "http://sensorml.com/ont/swe/property/RollAngle",
        "axisID": "X",
        "label": "Roll Angle",
        "uom": {
          "code": "deg"
        }
      }
    ]
  },
  "Sensor@iot.navigationLink": "/v1.0/Sensors(56)",
  "Thing@iot.navigationLink": "/v1.0/Things(1)"
}

A corresponding Observation could be encoded as a JSON array as it's currently done with MultiDatastream, implying values are provided in the same order as the coordinates in the Datastream:

{
  "@iot.id": "f4f87548eqqz",
  "phenomenonTime" : "2021-04-15T06:14:00.829Z",
  "resultTime" : "2021-04-15T06:14:00.829Z",
  "result" : [ 189.6, 1.2, -2.3 ],
  ...
}

or with a JSON object using names defined in the SWE Common result structure:

{
  "@iot.id": "f4f87548eqqz",
  "phenomenonTime" : "2021-04-15T06:14:00.829Z",
  "resultTime" : "2021-04-15T06:14:00.829Z",
  "result" : {
    "heading": 189.6,
    "pitch": 1.2,
    "roll": -2.3
  },
  ...
}

Note: that I have started working on a JsonSchema for SWE Common to support this

[liangsteve]

Discussion

One reason to have a navigational property (NavigationLink) rather than a property of an entity (or a property) is that there is a use case for linking back and forth, and NavigationLink is also very useful to encourage reusability and normalization.

For example, in this case when we change the ObservedProperty entity to a property of a resultType of a Datastream (as in the Example with scalar result), we lose the ability to link back from ObservedProperty to Datastreams. In this case, we can still have a navigational property between Datastream and ObservedProperty, as it is one-to-one relationship ("ObservedProperty@iot.navigationLink": "/v1.0/ObservedProperty(1)"), but in the case of Vector (e.g., Example with vector result), we will lose that ability to link back and forth. i.e., It would be difficult to search for "giving all Datastreams of that particular ObservedProperty".

[alexrobin]

Ok, I see. I think nothing prevents having both ObservedProperties navigation link at the root of the Datastream and a more detailed resultType (or resultSchema).

In the scalar case, it would be something like:

{
  "name": "Sensor 201 - Air Temperature",
  "description": "Datastream of temperature measurements",

  "Sensor@iot.navigationLink": "/v1.0/Sensors(201)",
  "Thing@iot.navigationLink": "/v1.0/Things(1)"
  "ObservedProperty@iot.navigationLink": "/v1.0/ObservedProperties(155)",

  "resultType": {
    "type": "Quantity",
    "definition": "http://mmisw.org/ont/cf/parameter/air_temperature",
    "label": "Air Temperature",
    "uom": {
        "code": "Cel",
        "symbol": "°C",
        "label": "Degree Celsius",
        "definition": "http://www.qudt.org/qudt/owl/1.0.0/unit/Instances.html#DegreeCelsius"
    }
  }
}

And in the vector case:

{
  "name": "Sensor 56 - Platform Attitude",
  "description": "Datastream of platform orientation measurements",

  "Sensor@iot.navigationLink": "/v1.0/Sensors(56)",
  "Thing@iot.navigationLink": "/v1.0/Things(1)",
  "ObservedProperties@iot.navigationLink": "/v1.0/Datastreams(2)/ObservedProperties",

  "resultType": {
    "type": "Vector",
    "definition": "http://www.opengis.net/def/property/OGC/0/PlatformOrientation",
    "referenceFrame": "http://www.opengis.net/def/cs/OGC/0/NED",
    "coordinates": [
      {
        "name": "heading",
        "type": "Quantity",
        "definition": "http://sensorml.com/ont/swe/property/TrueHeading",
        "axisID": "Z",
        "label": "Heading Angle",
        "uom": {
          "code": "deg"
        }
      },
      {
        "name": "pitch",
        "type": "Quantity",
        "definition": "http://sensorml.com/ont/swe/property/PitchAngle",
        "axisID": "Y",
        "label": "Pitch Angle",
        "uom": {
          "code": "deg"
        }
      },
      {
        "name": "roll",
        "type": "Quantity",
        "definition": "http://sensorml.com/ont/swe/property/RollAngle",
        "axisID": "X",
        "label": "Roll Angle",
        "uom": {
          "code": "deg"
        }
      }
    ]
  }
}

[hylkevds]

I like this a lot. It would also allow us to get rid of MultiDatastreams (See also #77)

Besides the obvious flexibility advantages this has over MDs , another big advantage is that we don't need the ordered-many-to-many-relation, as we need for MDs. Each ObservedProperty only needs to be linked once, even if it is used multiple times, and the order doesn't matter.

The DS-OP links could also be generated/managed by the server, making things easier for the user. Though for that to work we do have to be careful with how we define the contents of the definition fields. Currently the definition field of the ObservedProperty is not defined to be unique, and the server needs some way to find the OPs from what is in the definition field.

For the simple use-cases nothing would change, except that the DS-OP relation would become many-to-many.

[alexrobin]

@hylkevds Thanks for considering this. It is super important for us because we need to support more complex result types and provide proper metadata about it (i.e. beyond just a list of observables; you can see that in the vector example that also has a reference frame, etc).

I agree that an ObservedProperty 'definition' needs to be unique for this to work. Was there a use case for supporting multiple ObservedProperty resources with the same definition?

Let me know if I can be of help with this. I know I cannot attend the SWG telecons but I can contribute more asynchronously so it would be great if we can continue the conversation via GitHub.

[hylkevds]

I've also not seen real use cases with duplicate definition values for ObservedProperties, other than "laziness" when creating derived ObservedProperties... @KathiSchleidt What is your take on that?

Brainstorming a bit on other options:

• Using the self-link of the observedProperty. This directly points to the ObservedProperty and is unique. If we add an OData context to our responses this can be a relative link.
• Using the ID of the ObservedProperty
• Using custom entity linking

Using the OPs definition is probably the best option...

[KathiSchleidt]

@alexrobin I like the use case, worry about the creative bastardization of STA conventions! ;) Defining structures that bypass the data model, rely on consistent definitions in the DS & ObsProps seems like a way towards madness. I like Hylke's suggestion of just directly providing the self-link of the STA ObservedProperty, either in the definition (for now) or maybe one could extend the SWE-Common types with a link field.

On the JSON Encoding of SWE Common, what version of SWE Common is being utilized, both in your examples and underpinning the 17-011r2 BP? The UML I have for SWE Common seems to utilize UnitOfMeasure from ISO 19103, but that has a different naming convention. Is this documented somewhere?

[alexrobin]

@KathiSchleidt Yes I agree it's not ideal. I guess it depends how much you are willing to change things in the existing STA model vs. in the SWE Common model. My first example shows a ObservedProperty@iot.navigationLink in the component of the resultType like you suggest (this particular example is not a DataRecord but the same could be done within a DataRecord). I don't see any issue having such a link inside each component except that it is somewhat redundant with the definition property (and definition is a mandatory field in the SWE Common model).

I don't know how things were done in the Tasking extension, but I think the key resides in harmonizing the observation and the tasking side somehow. For instance, should we also define ControlledProperties that are the image of ObservedProperties, etc. like in SSN? If you do that then you'll have the same problem on the tasking side which uses SWE Common already...

The examples are made with SWE Common 2.0. The conceptual model refers to UnitOfMeasure of ISO 19103 but the XML or JSON encoding never includes such unit inlines. Units are always referenced in external registers or provided as a UCUM code. For reference, at some point during the development of the SWE standards, we were including units inline, and they followed the GML encoding.

[KathiSchleidt]

@alexrobin to my view, this exercise is flagging some of the issues we've had with SWE Common over the years, e.g., mandatory definition reference, but no other reference options. Are you planning to address issues various SWGs are having with SWE Common in the course of the planned revision, or will your focus be on the SensorHub requirements?

The question of how to provide the link STA to the ObservedProperties via relative links is closely related to our current conundrum pertaining to more general JSON encoding for OMS, as the nature of these links is tightly coupled with the underlying API. At least as long as you're extending STA, I believe it would be better to retain the wider OData logic behind this standard.

I've yet to use OGC SensorThings API Part 2 – Tasking Core, but agree that if modifications are made to the STA data model during the update to V2.0, these parts should stay aligned as they are in SOSA/SSN.

On UoM, thanks for the pointer to the trivialized encoding foreseen by GML. I am surprised that the requirement for a richer encoding more closely reflecting the 19103 model wasn't a topic during the previous SWE Common cycles. However, that still doesn't explain the following encoding that you've used above:

    "uom": {
        "code": "Cel",
        "symbol": "°C",
        "label": "Degree Celsius",
        "definition": "http://www.qudt.org/qudt/owl/1.0.0/unit/Instances.html#DegreeCelsius"
    }

[alexrobin]

@KathiSchleidt we can certainly address the definition issue at the SWE Common level and provide more reference options, aligning with other OGC JSON encodings in the process. Although it's already a URI/URL, which is what is commonly used everywhere else. What I don't want to do is tie SWE Common to OData specific patterns like @navigationLink.

However, I don't see any problem with STA slightly modifying the SWE Common JSON encoding to align it to OData patterns. Like you said, the exact syntax you use to implement resource linking is somewhat API dependent. So maybe the solution could be to just replace SWE Common definition by observedProperty@navigationLink... For me, this is just an encoding detail so it's fine as long as the mapping to the original model is clearly documented.

Regarding my extended uom example, I specifically put a comment in my example to say that the symbol and label properties are NOT currently in SWE Common. I was just playing around with ideas. In SWE Common we typically don't provide this info because we get it out-of-band, either from the UCUM database, or from the remote unit definition.

[KathiSchleidt]

@alexrobin on the SWE Common encoding you're utilizing here, I'm still a bit lost in finding the connection between the UML and XML representations of the SWE Common Data Model 2.0 and the JSON snippets you've been providing. Is there a mapping for this JSON encoding of SWE Common 2.0, or is this just freestyle?

[alexrobin]

The latest JSON schema for SWE Common is available here.

I think the examples I provided in this thread a long time ago should work with this schema.

The extra properties in the uom object are not currently in the JSON schema (we typically only provide the UCUM code or a URI. That's why I flagged them as "not in SWE Common but could be added in 2.1" in my example.

[humaidkidwai]

Can we consider using SWE Common to provide the result structure like what was done in the Tasking extension?

The latest JSON schema for SWE Common is available here.

I think the examples I provided in this thread a long time ago should work with this schema.

@alexrobin I am kind of lost in trying to find a mapping between the result structure you described and the SWE Common JSON schema you've shared. Could you elaborate on the mapping?

EDIT: I think you mean that the DataStream schema as defined in SWE Common (updated) can be adopted by DS (Datastream) Entity in STA to describe all the OP (ObservedProperty) and OBS (Observation) within the DS entity.

We still require a relationship between DS and OBS entities. The values: {} key can include NavigationLink to the related Observation(?) .

[humaidkidwai]

I also feel that with the SWE Common JSON Schema, we are repeating ourselves. We need to include NavigableProperties which store the same information. If we design the DS Entity to conform with SWE Common, we are going to waste a lot of memory by redundantly storing information in DS and OBS/OP

[hylkevds]

EDIT: I think you mean that the DataStream schema as defined in SWE Common (updated) can be adopted by DS (Datastream) Entity in STA to describe all the OP (ObservedProperty) and OBS (Observation) within the DS entity.

No, it's about adding a field to the STA Datastream that describes the format of the result field in the Observations of that Datastream. In 99% of our use cases we have a simple number value as Observation/result, and we don't need this. Only when we have an OM_ComplexObservation we need a way to describe what the result of these Observations look like, so that clients have any hope of being able to use the results.

That also means there is very little duplication. Only the ObservedProperty/definition string re-appears in the SWE.Common description and this can be in an n-to-m way. Each of the OPs linked to the Datastream can appear multiple times in the result description.