API does not return device metadata (update metadata not working)

[tstorek]

Hello,

I recognised that the endpoint to retrieve a device by id does not return the metadata added to attributes as documented in the nodelib docs.

This happens although the metadata is correctly posted to the create endpoint and the referenced entity is correctly initialised:

How to reproduce:

I sent this payload to the create endpoint:

"devices": [
    {
      "device_id": "device_with_meta",
      "service": null,
      "service_path": "/",
      "entity_name": "test_entity",
      "entity_type": "test_entity_type",
      "timezone": "Europe/Berlin",
      "timestamp": null,
      "apikey": "1234",
      "endpoint": null,
      "protocol": null,
      "transport": "HTTP",
      "lazy": [],
      "commands": [],
      "attributes": [
        {
          "name": "temperature",
          "type": "Number",
          "metadata": {
            "accuracy": {
              "type": "Text",
              "value": "+-5%"
            }
          },
          "expression": null,
          "entity_name": null,
          "entity_type": null,
          "reverse": null,
          "object_id": "temperature"
        }
      ],
      "static_attributes": [],
      "internal_attributes": null,
      "expressionLanguage": null,
      "explicitAttrs": false,
      "ngsiVersion": "v2"
    }
]

The created then entity looks like this:

    {
      "id": "test_entity",
      "type": "test_entity_type",
      "TimeInstant": {
        "type": "DateTime",
        "value": "2021-05-06T07:34:20.417Z",
        "metadata": {}
      },
      "temperature": {
        "type": "Number",
        "value": null,
        "metadata": {
          "accuracy": {
            "type": "Text",
            "value": "+-5%"
          }
        }
      }
    }

However, retrieving the device by its id returns this:

{
  "device_id": "device_with_meta",
  "service": "filip",
  "service_path": "/testing",
  "entity_name": "test_entity",
  "entity_type": "test_entity_type",
  "apikey": "1234",
  "transport": "HTTP",
  "lazy": [],
  "commands": [],
  "attributes": [
    {
      "name": "temperature",
      "type": "Number",
      "object_id": "temperature"
    }
  ],
  "static_attributes": [],
  "explicitAttrs": false,
  "ngsiVersion": "v2"
}

Am I missing an undocumented query parameter here or is it just a bug?

[AlvaroVega]

IMHO it is just a bug:

mettadata of atts are found in mongo device collection::

time=2021-09-30T09:04:25.296Z | lvl=DEBUG | corr=ffb60b27-472d-4191-93d8-83ef8b4226fc | trans=ffb60b27-472d-4191-93d8-83ef8b4226fc | op=IoTAgentNGSI.MongoDBDeviceRegister | from=n/a | srv=smartcity | subsrv=/ | msg=Device data found: {"_id":"6151d6756557af49325836a7","active":[{"name":"temperature","type":"Number","metadata":{"accuracy":{"type":"Text","value":"+1"}},"object_id":"temperature"}],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2021-09-27T14:34:29.748Z","id":"disp33a","type":"thing","name":"thing:disp33a","service":"smartcity","subservice":"/","internalId":null,"protocol":"IoTA-JSON","transport":"HTTP","polling":true,"explicitAttrs":false} | comp=IoTAgent

but are not returned by API:

curl -X GET 'http://localhost:4052/iot/devices/disp33a' -H 'fiware-service: smartcity' -H 'fiware-servicepath: /' {"device_id":"disp33a","service":"smartcity","service_path":"/","entity_name":"thing:disp33a","entity_type":"thing","transport":"HTTP","attributes":[{"object_id":"temperature","name":"temperature","type":"Number"}],"commands":[],"static_attributes":[],"protocol":"IoTA-JSON","explicitAttrs":false}

[fgalan]

Fixed by PR https://github.com/telefonicaid/iotagent-node-lib/pull/1113

@tstorek could you test again (using master branch code) and tell us if the issue has been solved, please? Thanks!

[tstorek]

@fgalan sorry for that late response. It seems to work just fine now. I tested with IoTA-JSON

{'libVersion': '2.18.0', 'port': '4041', 'baseRoot': '/', 'version': '1.20.0'}

However, updating the metadata are not correctly forwarded to the context broker.

I usually use 'put' for this.

I used this library for testing

https://github.com/RWTH-EBC/FiLiP/tree/master/tutorials/ngsi_v2

Exercise 7 should actually do that job. But I had to delete and add devices in order to create the correct entity representation.

[fgalan]

@tstorek I'm afraid I'm a bit confused. You say "It seems to work just fine now" but, at the same time, "updating the metadata are not correctly forwarded to the context broker".

Maybe do you mean that the issue described here is solved but a new issue has arisen? In that case, I'd suggest to create it a new and fresh issue for it (https://github.com/telefonicaid/iotagent-node-lib/issues/new/choose)

[tstorek]

@fgalan That is exactly what I tried to say. I was not sure if it is really a new issue because it seems so closely related to me.

Apparently, we can now create them properly on the one hand but cannot update them on the other. Are you sure that this a new issue.

Although metadata should be mostly static updating them once in a while should be possible. Especially, if additional metadata should be added to an existing attribute.

[fgalan]

@tstorek for the clarifications. I think we can keep this issue for it. I have slightly modified the tittle, to be a bit more precise on what's pending.

Could you provide the request you are using to update metadata (verb + url + headers + payload) which is not working, please?

[tstorek]

@fgalan here my curl commands

curl --location --request POST 'http://localhost:4041/iot/devices' \
--header 'fiware-service: tutorial' \
--header 'fiware-servicepath: /your_path' \
--header 'Content-Type: application/json' \
--data-raw '{
    "devices": [
        {
            "timezone": "Europe/London",
            "apikey": "your_path",
            "protocol": "IoTA-JSON",
            "transport": "MQTT",
            "explicitAttrs": false,
            "device_id": "device:001",
            "service": "filip_tutorial",
            "service_path": "/your_path",
            "entity_name": "urn:ngsi-ld:WeatherStation:001",
            "entity_type": "WeatherStation",
            "lazy": [],
            "commands": [],
            "attributes": [
                {
                    "name": "temperature",
                    "type": "Number",
                    "metadata": {
                        "unitCode": {
                            "type": "Text",
                            "value": "CEL"
                        }
                    },
                    "object_id": "t_amb"
                }
            ],
            "static_attributes": [],
            "internal_attributes": [],
            "ngsiVersion": "v2"
        }
    ]
}'

This works and the iot-agent returns the right configuration and creates the entities in orion:

curl --location --request GET 'http://localhost:4041/iot/devices/device:001' \
--header 'fiware-service: tutorial' \
--header 'fiware-servicepath: /your_path' \
--data-raw ''

returns

{
    "device_id": "device:001",
    "apikey": "your_path",
    "service": "tutorial",
    "service_path": "/your_path",
    "entity_name": "urn:ngsi-ld:WeatherStation:001",
    "entity_type": "WeatherStation",
    "transport": "MQTT",
    "attributes": [
        {
            "object_id": "t_amb",
            "name": "temperature",
            "type": "Number",
            "metadata": {
                "unitCode": {
                    "type": "Text",
                    "value": "CEL"
                },
                "accuracy": {
                    "type": "Text",
                    "value": "5%"
                }
            }
        }
    ],
    "lazy": [],
    "commands": [],
    "static_attributes": [],
    "internal_attributes": [],
    "protocol": "IoTA-JSON",
    "explicitAttrs": false,
    "ngsiVersion": "v2"
}

and for orion:

curl --location --request GET 'http://localhost:1026/v2/entities/urn:ngsi-ld:WeatherStation:001' \
--header 'fiware-service: tutorial' \
--header 'fiware-servicepath: /your_path' \
--data-raw ''

response:

{
    "id": "urn:ngsi-ld:WeatherStation:001",
    "type": "WeatherStation",
    "TimeInstant": {
        "type": "DateTime",
        "value": "2022-02-16T10:57:27.470Z",
        "metadata": {}
    },
    "temperature": {
        "type": "Number",
        "value": null,
        "metadata": {
            "unitCode": {
                "type": "Text",
                "value": "CEL"
            }
        }
    }
}

Now, I try to add additional metadata via updating the device configuration!

curl --location --request PUT 'http://localhost:4041/iot/devices/device:001' \
--header 'fiware-service: tutorial' \
--header 'fiware-servicepath: /your_path' \
--header 'Content-Type: application/json' \
--data-raw '        {
            "lazy": [],
            "commands": [],
            "attributes": [
                {
                    "name": "temperature",
                    "type": "Number",
                    "metadata": {
                        "unitCode": {
                            "type": "Text",
                            "value": "CEL"
                        },
                        "accuracy": {
                            "type": "Text",
                            "value": "5%"
                        }
                    },
                    "object_id": "t_amb"
                }
            ],
            "static_attributes": [],
            "internal_attributes": []
        }'

afterwards the iot-agent returns:

{
    "device_id": "device:001",
    "apikey": "your_path",
    "service": "tutorial",
    "service_path": "/your_path",
    "entity_name": "urn:ngsi-ld:WeatherStation:001",
    "entity_type": "WeatherStation",
    "transport": "MQTT",
    "attributes": [
        {
            "object_id": "t_amb",
            "name": "temperature",
            "type": "Number",
            "metadata": {
                "unitCode": {
                    "type": "Text",
                    "value": "CEL"
                },
                "accuracy": {
                    "type": "Text",
                    "value": "5%"
                }
            }
        }
    ],
    "lazy": [],
    "commands": [],
    "static_attributes": [],
    "internal_attributes": [],
    "protocol": "IoTA-JSON",
    "explicitAttrs": false,
    "ngsiVersion": "v2"
}

but Orion sticks with the former entity:

{
    "id": "urn:ngsi-ld:WeatherStation:001",
    "type": "WeatherStation",
    "TimeInstant": {
        "type": "DateTime",
        "value": "2022-02-16T10:57:27.470Z",
        "metadata": {}
    },
    "temperature": {
        "type": "Number",
        "value": null,
        "metadata": {
            "unitCode": {
                "type": "Text",
                "value": "CEL"
            }
        }
    }
}

Hence, this should be fixed! Additionally, I noticed that updating a device configuration accepts also nested metadata such as (I know that it is not recommended):

{
                    "name": "temperature",
                    "type": "Number",
                    "metadata": {
                        "unitCode": {
                            "type": "Text",
                            "value": "CEL"
                        },
                        "accuracy": {
                            "type": "Number",
                            "value": "5",
                            "metadata": {
                                "unitCode": {
                                    "type": "TEXT",
                                    "value": "P1"
                                }
                            }
                        }
                    },
                    "object_id": "t_amb"
                }

But posting it in the first place leads to an ERROR (incl. I typo ;)):

{
    "name": "ENTITY_GENERIC_ERROR",
    "message": "Error accesing entity data for device: device:001 of type: WeatherStation"
}

From my point of view this is inconsistent. Furthermore, the IoT-Agent should not limit Orion's functionalities in any way. Hence, although nested metadata is not recommended in Orion's specification it might be still valuable in some situations because adding a static attribute for the accuracy of the temperature attribute seems especially for multi-sensors seems to be non-sense without unit information.

Thank you!

[fgalan]

So it seems we have two problems here:

1. Metadata are not updated when using PUT /iot/devices/{devId} operation
2. Trying to use a nested metadata (i.e. metadata which value is a JSON Object) in POST /iot/devices causes error
   • Maybe the same problem with metadata which value is a JSON Array?

Is my understanding correct?

[tstorek]

@fgalan

1. absolutely
2. is there another open issue about that. But yes nested metadata vs JSON array vs structuredValue does not sound like big difference.

[fgalan]

@tstorek thanks for your feedback!

By the moment this issue is not in our priorities, but it would be great if somebody in the community would like to have a look and solve it. We could provide support to that effort reviewing the pull request with the fix and giving feedback on it :)