telefonicaid / iotagent-node-lib

Module to enable IoT Agent developers to build custom agents for their devices that can easily connect to NGSI Context Brokers
https://iotagent-node-lib.rtfd.io/
GNU Affero General Public License v3.0
60 stars 87 forks source link

Wrong association of measurements to the correct device using different service groups but same device_id #1589

Closed tudamp closed 6 months ago

tudamp commented 7 months ago

IoT Agent Node Lib version the issue has been seen with

4.1.0

Bound or port used (API interaction)

Southbound (Devices data API)

NGSI version

NGSIv2

Are you running a container?

Yes, I am using a contaner (Docker, Kubernetes...)

Image type

distroless

Expected behaviour you didn't see

By creating an environment using service groups with these settings: Parameters Type1:Sensor01 Type2:Sensor01
Fiware-Service service1 service1
Fiware-ServicePath / /
Resource /iot/json /iot/json
API Key apikey01 apikey02
Entity Type Type1 Type2
Device ID Sensor01 Sensor01
Entity Name Type1:Sensor01 Type2:Sensor01

By sending the measurements first of Type1:Sensor01 with apikey1 and then of Type2:Sensor01 with its apikey2, the agent should:

  1. recognise the service group of the first send (using apikey01 and resource), create the device if not present with the correct Entity type of the service group, send the measurement to cb

  2. repeat the same behaviour with the second data send from the Type2:Sensor01

Unexpected behaviour you saw

The step1:

  1. recognise the service group of the first send (using apikey01 and resource), create the device if not present with the correct Entity type of the service group, send the measurement to cb

occurs correctly: the service group matched to the apikey is recognised and the settings (type1) combined with the measurement arriving from the device.

In step 2, the service group matched to apikey2 is recognised, but given the presence of a device with the same id, the measurement is combined with this device, resulting in an incorrect match with the entity in the cb (type1 instead of type2). The device in the second service group is never created.

Steps to reproduce the problem

No response

Configs

environment:
    - "IOTA_CB_HOST=orion"
    - "IOTA_CB_PORT=1026"
    - "IOTA_NORTH_PORT=4041"
    - "IOTA_REGISTRY_TYPE=mongodb"
    - "IOTA_MONGO_HOST=mongodb"
    - "IOTA_MONGO_PORT=27017"
    - "IOTA_MONGO_DB=iotagent-json"
    - "IOTA_HTTP_PORT=7896"
    - "IOTA_CB_NGSI_VERSION=v2"
    - "IOTA_DEFAULT_RESOURCE=/iot/json"
    - "IOTA_FALLBACK_TENANT=openiot"
    - "IOTA_PROVIDER_URL=http://iot-agent:4041"

Log output

time=2024-03-12T16:59:14.440Z | lvl=DEBUG | corr=e97d3ce3-fad3-4cd2-b3fe-0e75dd246c35 | trans=e97d3ce3-fad3-4cd2-b3fe-0e75dd246c35 | op=IoTAgentNGSI.DomainControl | from=n/a | srv=n/a | subsrv=n/a | msg=response-time: 1 | comp=IoTAgent
time=2024-03-12T16:59:18.521Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IOTAJSON.HTTP.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Using body { attr1: 2, location: '9,100' } | comp=IoTAgent
time=2024-03-12T16:59:18.522Z | lvl=DEBUG | corr=fbb05258-b61b-48d2-a97e-b5d561809444 | trans=fbb05258-b61b-48d2-a97e-b5d561809444 | op=IOTAJSON.HTTP.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Parsed data array: [{"attr1":2,"location":"9,100"}] | comp=IoTAgent
time=2024-03-12T16:59:18.522Z | lvl=DEBUG | corr=fbb05258-b61b-48d2-a97e-b5d561809444 | trans=fbb05258-b61b-48d2-a97e-b5d561809444 | op=IOTAJSON.HTTP.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Processing multiple HTTP measures for device Sensor01 with apiKey 49EbDeCd5A0228CB7Be30FAf664DE8ba3943d27 | comp=IoTAgent
time=2024-03-12T16:59:18.522Z | lvl=DEBUG | corr=fbb05258-b61b-48d2-a97e-b5d561809444 | trans=fbb05258-b61b-48d2-a97e-b5d561809444 | op=IoTAgentNGSI.MongoDBGroupRegister | from=n/a | srv=n/a | subsrv=n/a | msg=Looking for group params ["resource","apikey"] with queryObj {"resource":"/iot/json","apikey":"49EbDeCd5A0228CB7Be30FAf664DE8ba3943d27"} | comp=IoTAgent
time=2024-03-12T16:59:18.524Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBGroupRegister | from=n/a | srv=service1 | subsrv=/ | msg=Device group data found: {"_id":"65f084547f160806af577e2d","attributes":[{"name":"attr2","object_id":"attr2","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"resource":"/iot/json","apikey":"49EbDeCd5A0228CB7Be30FAf664DE8ba3943d27","type":"Type2","service":"service1","subservice":"/","cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.524Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | from=n/a | srv=service1 | subsrv=/ | msg=Looking for device with id [Sensor01] and queryParams [{"id":"Sensor01","service":"service1","subservice":"/","apikey":"49EbDeCd5A0228CB7Be30FAf664DE8ba3943d27"}]. | comp=IoTAgent
time=2024-03-12T16:59:18.525Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | from=n/a | srv=service1 | subsrv=/ | msg=Device [Sensor01] not found. | comp=IoTAgent
time=2024-03-12T16:59:18.525Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | from=n/a | srv=service1 | subsrv=/ | msg=Looking for device with id [Sensor01] and queryParams [{"id":"Sensor01","service":"service1","subservice":"/"}]. | comp=IoTAgent
time=2024-03-12T16:59:18.527Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.MongoDBDeviceRegister | from=n/a | srv=service1 | subsrv=/ | msg=Device data found: {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D"} | comp=IoTAgent
time=2024-03-12T16:59:18.527Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.DeviceService | from=n/a | srv=n/a | subsrv=n/a | msg=deviceData before merge with conf: {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D"} defaults: [null,null,[],[],[],[],[]] fields: ["lazy","active","staticAttributes","commands","subscriptions"] configuration {"_id":"65f084547f160806af577e2d","attributes":[{"name":"attr2","object_id":"attr2","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"resource":"/iot/json","apikey":"49EbDeCd5A0228CB7Be30FAf664DE8ba3943d27","type":"Type2","service":"service1","subservice":"/","cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.527Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.DeviceService | from=n/a | srv=n/a | subsrv=n/a | msg=deviceData after merge with conf: {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","active":[{"name":"attr2","object_id":"attr2","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.527Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IOTAJSON.HTTP.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Parsing payloadDataArr [{"attr1":2,"location":"9,100"}] for device {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","active":[{"name":"attr2","object_id":"attr2","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.527Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentJSON.commonBinding | from=n/a | srv=service1 | subsrv=/ | msg=extractAttributes current {"attr1":2,"location":"9,100"} payloadType undefined | comp=IoTAgent
time=2024-03-12T16:59:18.528Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IOTAJSON.HTTP.Binding | from=n/a | srv=n/a | subsrv=n/a | msg=Processing measure device Type1:Sensor01 with attributeArr [[{"name":"attr1","type":"string","value":2},{"name":"location","type":"geo:point","value":"9,100"}]] attributeValues [{"name":"attr1","type":"string","value":2},{"name":"location","type":"geo:point","value":"9,100"}] | comp=IoTAgent
time=2024-03-12T16:59:18.528Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.NGSIService | from=n/a | srv=service1 | subsrv=/ | msg=executeWithDeviceInfo entityName Type1:Sensor01 type Type1 apikey  attributes [{"name":"attr1","type":"string","value":2},{"name":"location","type":"geo:point","value":"9,100"}] deviceInformation {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","active":[{"name":"attr2","object_id":"attr2","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.528Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.MongoDBGroupRegister | from=n/a | srv=n/a | subsrv=n/a | msg=Looking for group params ["type"] with queryObj {"type":"Type1"} | comp=IoTAgent
time=2024-03-12T16:59:18.533Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.MongoDBGroupRegister | from=n/a | srv=service1 | subsrv=/ | msg=Device group data found: {"_id":"65f061ac7f1608515d577dbc","attributes":[{"name":"attr1","object_id":"attr1","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"resource":"/iot/json","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","type":"Type1","service":"service1","subservice":"/","cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.533Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=service1 | subsrv=/ | msg=sendUpdateValueNgsi2 called with: entityName=Type1:Sensor01,  measures={"attr1":2,"location":"9,100"},  typeInformation={"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","active":[{"name":"attr2","object_id":"attr2","type":"Number"},{"name":"location","object_id":"location","type":"geo:point"}],"cbHost":"http://orion:1026"}, initial jexlContext={"attr1":2,"location":"9,100","id":"Sensor01","type":"Type1","service":"service1","subservice":"/","entity_name":"Type1:Sensor01","TimeInstant":"2024-03-12T16:59:18.533Z"}, timestamp=true with value="2024-03-12T16:59:18.533Z" | comp=IoTAgent
time=2024-03-12T16:59:18.534Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.NGSIUtils | from=n/a | srv=service1 | subsrv=/ | msg=typeInformation {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","active":[{"name":"attr2","object_id":"attr2","type":"Number","skipValue":null,"hitted":false,"value":null,"metadata":{"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}},{"name":"location","object_id":"location","type":"geo:point","skipValue":null,"hitted":true,"value":"9,100","metadata":{"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}}],"cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.534Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.NGSIUtils | from=n/a | srv=service1 | subsrv=/ | msg=typeInformation {"_id":"65f063037f16083bb1577dcd","lazy":[],"commands":[],"staticAttributes":[],"subscriptions":[],"creationDate":"2024-03-12T14:13:23.656Z","id":"Sensor01","type":"Type1","name":"Type1:Sensor01","service":"service1","subservice":"/","apikey":"CE7a8df3367A9E66cDCaf1eB516b6C6FBFFc08D","active":[{"name":"attr2","object_id":"attr2","type":"Number","skipValue":null,"hitted":false,"value":null,"metadata":{"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}},{"name":"location","object_id":"location","type":"geo:point","skipValue":null,"hitted":true,"value":"9,100","metadata":{"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}}],"cbHost":"http://orion:1026"} | comp=IoTAgent
time=2024-03-12T16:59:18.534Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=service1 | subsrv=/ | msg=Updating device value in the Context Broker at: "http://orion:1026/v2/entities?options=upsert" | comp=IoTAgent
time=2024-03-12T16:59:18.534Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=service1 | subsrv=/ | msg=Using the following NGSI v2 request: {"url":"http://orion:1026/v2/entities?options=upsert","method":"POST","headers":{"fiware-service":"service1","fiware-servicepath":"/"},"json":{"id":"Type1:Sensor01","type":"Type1","location":{"type":"geo:point","value":"9,100","metadata":{"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}},"attr1":{"type":"string","value":2,"metadata":{"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}},"TimeInstant":{"type":"DateTime","value":"2024-03-12T16:59:18.533Z"}}} | comp=IoTAgent
time=2024-03-12T16:59:18.534Z | lvl=DEBUG | corr=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | trans=5cdb5cb0-f5b9-477d-bc72-04ac4206075e | op=IoTAgentNGSI.Request | from=n/a | srv=service1 | subsrv=/ | msg=Options: {
    "url": "http://orion:1026/v2/entities?options=upsert",
    "method": "POST",
    "headers": {
        "fiware-service": "service1",
        "fiware-servicepath": "/"
    },
    "json": {
        "id": "Type1:Sensor01",
        "type": "Type1",
        "location": {
            "type": "geo:point",
            "value": "9,100",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-03-12T16:59:18.533Z"
                }
            }
        },
        "attr1": {
            "type": "string",
            "value": 2,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2024-03-12T16:59:18.533Z"
                }
            }
        },
        "TimeInstant": {
            "type": "DateTime",
            "value": "2024-03-12T16:59:18.533Z"
        }
    }
} | comp=IoTAgent
AlvaroVega commented 7 months ago

Not sure if is a corner case or what should be the right behaviour. At the second step device named Sensor01 with an entityType and entityName is already found and then used.

If your add an static attribute to each group then the resulting entity has both static attributes.

Maybe the bug is that at second step Sensor01 device should be updated with new provided entityName and entityType?

AlvaroVega commented 7 months ago

Maybe (or not) before second step (send second measure) you can update provisioned device to ensure entityName and entityType are updated on Sensor01 device and then send the measure.

tudamp commented 7 months ago

Hi Alvaro, thank you for your reply.

According to the documentation about service groups: They are identified by an apikey and a resource and mapped to a particular entity type. [...] For every config group, the pair (resource, apikey) must be unique (as it is used to identify which group to assign to which device).

From my point of view, once a measurement is 'identified' with a specific apikey and resource, it cannot be mapped to a different entity that belongs to a different group. Probably the main problem is that the agent seems to ignore the entity type, which is instead important when talking to the context broker.

There are probably two solutions:

The second solution, however, seems to me to contradict what is written in the documentation.

AlvaroVega commented 7 months ago

Currently is not possible provision two devices in the same service/subservice with the same id but different apikey. And apikey at device level is mandatory since not along ago

tudamp commented 7 months ago

Perhaps this should be made clearer in the documentation, because at present it seems to describe a different behaviour (at least in my interpretation of the text).

The problem I see is that the definition of uniqueness within the agent and the CB is different. This causes confusion as there is a 1:1 logical mapping between the device in the iot agent and the CB. Do you think it makes sense to also include the device type in the uniqueness definition to standardise the behaviour with the CB?

Regardless of these considerations in the roadmap, given the current definition of uniqueness, I think the current behaviour may not be correct and the second measurement should be rejected.

fgalan commented 7 months ago

Perhaps this should be made clearer in the documentation, because at present it seems to describe a different behaviour (at least in my interpretation of the text).

Maybe you could do a pull request with a proposal of change to the documentation to make things clearer, based in your experience with this issue, please? Thanks in advance!

fgalan commented 7 months ago

Do you think it makes sense to also include the device type in the uniqueness definition to standardise the behaviour with the CB?

As far as I understand "type" is a concept used in CB associated to entities but it doesn't have an equivalent at device level at IOTA. In sum:

The mapping between both worlds (entity_id+entity_type <-> device_id) is done by IOTA, either by explicit provision or by autoprovision.

Please @AlvaroVega provide feedback if I'm wrong.

AlvaroVega commented 7 months ago

entity_name and entity_type are fields of device to link a device with a CB entity (which has id ant type), but are not a key in Devices model and typically are inhered from Group (as apikey) when autoprovision.

fgalan commented 7 months ago

PR https://github.com/telefonicaid/iotagent-node-lib/pull/1592 should fix this issue, allowing several devices with same device_id in the same service and subservice (as long as they are using different apikeys).

@tudamp could you test again, please? You can get the latest iotagent-json docker container from dockerhub at telefonicaiot/iotagent-json:latest. Thanks in advance!

fgalan commented 6 months ago

Release 4.4.0 including this fix has been released today, so we are going to close this issue.

@tudamp of course it can reopened if needed, if you provided feedback at the end.