Measurement type is static as "ThinEdgeMeasurement"

[mbay-ODW]

Is your feature request related to a problem? Please describe. The Domain Model of Measurements within Cumulocity gives three degrees of freedom for storing measurements:

1. Measurment Fragment
2. Measurement Series
3. Measurement Type

As I do understand all those three columns are indexed within the mongoDB such that it is recommended to model data in a way that all those 3 informations somehow are considered for the measurements to perform fast queries. Why is the 3. entry of type not used within thinedge properly. However I do understand that it makes sense to have the information of "ThinEdgeMeasurement" as an information on the measurement itself but have the feeling that it should not be the type. What are your thoughts on that?

Describe the solution you'd like If something such as a type exists within the initial payload on the local MQTT broker the mapper should use that. Static types could be used if nothing such as type is provided in the payload.

[simeonthefirst]

Also discussed in #648

[reubenmiller]

Is your feature request related to a problem? Please describe.

thin-edge.io measurement creation has the following limitations when using the custom json payload format

• type field is reserved which means the user cannot set their own value to help classify the measurements
• thin-edge does not allow adding additional fragments to the measurement body. Custom fragments are often used to add additional context to the measurement using free-form data (e.g. this additional info can be accessed during event processing to control how the measurement is processed)

These limitations result in a sub-optimal experience when using Cumulocity IoT as the full Cumulocity IoT api cannot be utilised to query and process the measurements.

Example 1: Measurement with a user-defined type value

The following measurement payload sets the type field so that the user can mark specific fragments with additional meta information which helps classify the type of measurement.

tedge mqtt pub tedge/measurements/custom_child_device '{ "current": {"L1": 9.5, "L2": 1.3}, "type": "MySpecialType" }'

Current the type field is fixed, so the tedge-mapper-c8y throws the following error:

ERROR tedge-mapper-c8y:messages{ops_dir=Some("/etc/tedge/operations/c8y")}: tedge_mapper::core::converter: Mapping error: Invalid JSON: Invalid measurement name: "type" is a reserved word. at line 1 column 43: `": "MySpecialType", "customValue": "Some other info" }

Example 2: Measurement with user-defined custom fragments

The following example shows the user adding the customInfo field to the fragment. The example only shows a string example, however the fragments can be of any valid json data type, e.g. object, array, boolean etc.

tedge mqtt pub tedge/measurements/custom_child_device '{ "current": {"L1": 9.5, "L2": 1.3}, "customInfo": "Some other info" }'

The above command invokes the following error in tedge-mapper-c8y

ERROR tedge-mapper-c8y:messages{ops_dir=Some("/etc/tedge/operations/c8y")}: tedge_mapper::core::converter: Mapping error: Invalid JSON: invalid type: string "Some other info", expected ThinEdge single or multi-value measurement at line 1 column 68: `" }

Describe the solution you'd like

• If the user specifies a non-empty type value, then the measurement should be created in Cumulocity IoT using the user-defined value. If the type field is not present, or it is empty, then the default value ThinEdgeMeasurement should be used
• Custom fragments (e.g. non-value fragments) should be allowed in the payload and included in the measurement when it is sent to Cumulocity IoT

The examples in the previous section should result in the measurement creation with the custom type field and custom fragments.

Describe alternatives you've considered

No alternatives were considered as the suggested solution is the standard implementation when writing a custom agent.

[PradeepKiruvale]

Can there be multiple custom fragments as part of the measurement message?

[didier-wenzek]

As highlighted by https://github.com/thin-edge/thin-edge.io/pull/1911#discussion_r1164216090, this issue needs some clarifications.

• Where can be defined a measurement type?
  • Only at the top level of a measurement message as in `{ "type": "MyMeasurement", "location": { "latitude": 32.54, "longitude": -117.67, "altitude": 98.6 }}?
  • Inside a group of measurements as in { "location": { "latitude": 32.54, "longitude": -117.67, "altitude": 98.6, "type": "MyMeasurement" }}?
  • Are the two previous messages equivalent? i.e. translated to the same Cumulocity Json Message?
• Where custom fragments can be added?
  • At any level? as in { "location": { "latitude": 32.54, "longitude": -117.67, "altitude": 98.6, "quality": "Uncertain" }}
• How deep can be a custom fragment?
  • Is this anything not understood by the current implementation?
• How custom fragments are translated into Cumulocity Json?
  • { "temperature": 23.0, "pressure": 220.0 } is translated into {"type": "ThinEdgeMeasurement", "time": "2021-04-08T00:00:00+05:00", "temperature": {"temperature": {"value": 23.0}}, "pressure": {"pressure": {"value": 220.0}}}
  • How should { "temperature": 23.0, "pressure": 220.0 , "quality": "Uncertain" } be translated?

[reubenmiller]

The measurement type is only applicable for Cumulocity measurements, and it can only be defined on the root level, as per the following example:

{
    "type": "MyMeasurement",    // type should not be known in the tedge measurement format, it is just a non-value fragment
    "location": {
        "latitude": 32.54,
        "longitude": -117.67,
        "altitude": 98.6
    }
}

Note: The type should not be known in the tedge measurement format, it is just a non-value and should be passed on by default to the appropriate cloud mapper without interpreting what it is. The cloud mapper is then responsible for using the value and applying a default value (if it is needed).

The above tedge measurement would then be mapped to a Cumulocity measurement as follows:

{
    "type": "MyMeasurement",    // type allows user to further group measurements
    "location": {        // location is the `valueFragmentType` in c8y terminology
        "latitude": {    // latitude is the `valueFragmentSeries` in c8y terminology
            "value": 32.54,
            "unit": ""
        },
        "longitude": {    // longitude is the `valueFragmentSeries` in c8y terminology
            "value": -117.67,
            "unit": ""
        },
        "altitude": {
            "value": 98.6,
            "unit": ""
        }
    }
}

The type fragment is normally used by Cumulocity users to further classify/group types of measurements. The Cumulocity REST API allows users to query for measurements based on the type, valueFragmentType and valueFragmentSeries. This enables the users to be more flexible on how data is recalled and permission rules are applied.

[reubenmiller]

If it becomes difficult for tedge to detect which is is the measurement values (e.g. numeric values) and which is the meta info (such as type), then it might be more useful to use a reserved fragment/property name which stores all meta info, for example:

{
    "meta" : {  // 'meta' would be a reserved fragment/property name which holds all meta data (e.g. non-value data)
        "type": "MyMeasurement",
        "otherCustomData": {
            "something": {
                "foo": "bar"
            }
        }
    },
    "location": {
        "latitude": 32.54,
        "longitude": -117.67,
        "altitude": 98.6
    }
}

It would be up to each cloud mapper to decide what it does with the meta data...it could either ignore it, or pick out specific parts of it.

[gligorisaev]

checked