telefonicaid / iotagent-node-lib

Module to enable IoT Agent developers to build custom agents for their devices that can easily connect to NGSI Context Brokers
https://iotagent-node-lib.rtfd.io/
GNU Affero General Public License v3.0
60 stars 85 forks source link

Add tests for ID and Type colisions #1482

Closed mapedraza closed 10 months ago

AlvaroVega commented 10 months ago

With this branch the fix of https://github.com/telefonicaid/iotagent-node-lib/pull/1483 I'm getting some errors:

707 passing (41s) 11 pending 3 failing

1) NGSI-v2 - Active attributes test When the IoT Agent receives new information from a device and CBis defined using environment variables should change the value of the corresponding attribute in the context broker: Error: Timeout of 8000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/avega/tme/fiware/iotagent-node-lib/test/unit/ngsiv2/ngsiService/active-devices-test.js) at listOnTimeout (node:internal/timers:564:17) at process.processTimers (node:internal/timers:507:7)

2) NGSI-v2 - Active attributes test When the IoT Agent receives autoprovisioned id and type measures should not affect to the real ID and Type to store in the context broker: Uncaught AssertionError: expected RequestError { name: 'RequestError', code: 'ENOTFOUND', timings: Object { start: 1696262045323, socket: 1696262045323, lookup: 1696262045325, connect: undefined, secureConnect: undefined, upload: undefined, response: undefined, end: undefined, error: 1696262045325, abort: undefined, phases: Object { wait: 0, dns: 2, tcp: undefined, tls: undefined, request: undefined, firstByte: undefined, download: undefined, total: 2 } }, message: 'getaddrinfo ENOTFOUND cbhost' } to not exist at /home/avega/tme/fiware/iotagent-node-lib/test/unit/ngsiv2/ngsiService/active-devices-test.js:958:28 at /home/avega/tme/fiware/iotagent-node-lib/lib/services/ngsi/entities-NGSI-v2.js:86:497 at /home/avega/tme/fiware/iotagent-node-lib/lib/request-shim.js:64:728 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

3) NGSI-v2 - Active attributes test When the IoT Agent receives provisioned id and type measures should not affect to the real ID and Type to store in the context broker: Error: Timeout of 8000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/avega/tme/fiware/iotagent-node-lib/test/unit/ngsiv2/ngsiService/active-devices-test.js) at listOnTimeout (node:internal/timers:564:17) at process.processTimers (node:internal/timers:507:7)

mapedraza commented 10 months ago

3 Cases:

Case 1 Receiving measures (id and type) not provisioned (autoprovision)

In this case, overwrites the id and type values with the measured received. id and type are converted to JSON objects

{
        "id": {
            "value": "idIoTA",
            "type": "text"
        },
        "type": {
            "value": "typeIoTA",
            "type": "text"
        },
        "meas": {
            "value": "measIoTA",
            "type": "String"
        }
    }

Case 2. Mapping measures (i and t) to id and type attributes

In this case, it ignores the mappings, working as expected

Case 3. object_id = id and type

In this case, it ignores id and type, and they are not present in the request.

[{
    "url": "http://192.168.1.1:1026/v2/op/update",
    "method": "POST",
    "headers": {
        "fiware-service": "smartgondor",
        "fiware-servicepath": "gardens"
    },
    "json": {
        "entities": [
            {
                "meas": {
                    "value": "measIoTA",
                    "type": "String"
                }
            }
        ],
        "actionType": "append"
    }
}]
mapedraza commented 10 months ago

I suggest to change this PR destination to branch fix/remove_measures_with_attribute_name_id_type (PR #1483)

mapedraza commented 10 months ago

With this branch the fix of #1483 I'm getting some errors:

707 passing (41s) 11 pending 3 failing

@AlvaroVega After https://github.com/telefonicaid/iotagent-node-lib/pull/1482/commits/7197f627e5d522e046d249a005cc83ed1b137c05

Only 1 test is failing with your branch code (case 3). (last commit chaged order and added some minor fixes to the tests)

nyc --reporter=text mocha --recursive test/unit/ngsiv2/ngsiService/active-devices-test.js --reporter spec --timeout 8000 --ui bdd --exit --color true
{"op":"IoTAgentNGSI.JEXL","time":"2023-10-02T16:16:14.548Z","lvl":"INFO","msg":"Trasformations can be added to JEXL parser"}

(node:28928) [DEP0097] DeprecationWarning: Using a domain property in MakeCallback is deprecated. Use the async_context variant of MakeCallback or the AsyncResource class instead. (Triggered by calling processImmediate on process.)
(Use `node --trace-deprecation ...` to show where the warning was created)
  NGSI-v2 - Active attributes test
    When the IoT Agent receives new information from a device
      ✔ should change the value of the corresponding attribute in the context broker
    When the IoT Agent receives new information and the timestamp flag is on
      ✔ should add the timestamp to the entity and all the attributes
    When the IoT Agent receives new information and the timestamp flag is false
      ✔ should not add the timestamp to the entity and the attributes
    When the IoT Agent receives new information and the timestamp flag is false but the measure contains timeInstant
      ✔ should add the timestamp attribute to the entity but not as attribute metadata
    When the IoTA gets a set of values with a TimeInstant which are not in ISO8601 format
      ✔ should fail with a 400 BAD_TIMESTAMP error
    When the IoTA gets a set of values with a TimeInstant which are in ISO8601 format without milis
      ✔ should not fail
    When the IoT Agent receives new information, the timestamp flag is on and timezone is defined
      ✔ should add the timestamp to the entity and all the attributes
    When the IoTA gets a set of values with a TimeInstant and the timestamp flag is on
      ✔ should not override the received instant and should add metadatas for this request
    When the IoTA gets a set of values with a TimeInstant, the timestamp flag is onand timezone is defined
      ✔ should not override the received instant and should not add metadatas for this request
    When the IoT Agent receives information from a device whose type doesn't have a type name
      ✔ should fail with a 500 TYPE_NOT_FOUND error
    When the Context Broker returns an HTTP error code updating an entity
      ✔ should return ENTITY_GENERIC_ERROR an error to the caller
    When the Context Broker returns an application error code updating an entity
      ✔ should return ENTITY_GENERIC_ERROR an error to the caller
    When there is a transport error connecting to the Context Broker
      ✔ should return a ENTITY_GENERIC_ERROR error to the caller
    When the IoT Agent recieves information for a type with a configured Context Broker
      ✔ should use the Context Broker defined by the type
    When an IoT Agent receives information for a type with static attributes
      ✔ should decorate the entity with the static attributes
    When an IoT Agent receives information for a type with static attributes with metadata
      ✔ should decorate the entity with the static attributes
    When the IoT Agent receives new information from a device
      ✔ should change the value of the corresponding attribute in the context broker
    When the IoT Agent receives autoprovisioned id and type measures
      ✔ should not affect to the real ID and Type to store in the context broker
    When the IoT Agent receives provisioned id and type measures with different object_id names
      ✔ should not affect to the real ID and Type to store in the context broker
    When the IoT Agent receives provisioned id and type measures with the same object_id name
      1) should not affect to the real ID and Type to store in the context broker
    When the IoT Agent receives new information from a device and CBis defined using environment variables
      ✔ should change the value of the corresponding attribute in the context broker

  20 passing (415ms)
  1 failing

  1) NGSI-v2 - Active attributes test
       When the IoT Agent receives provisioned id and type measures with the same object_id name
         should not affect to the real ID and Type to store in the context broker:

Being the payload of the request:

time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 idTypeSSS are [{"name":"id","type":"String"},{"name":"type","type":"String","value":"StupidDevice2"},{"name":"service","type":"String"},{"name":"subservice","type":"String"},{"name":"entity_name","type":"String"}] 
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 pre-initial non-explicitAttrs payload={"entities":[{"id":"stupiddevice3","type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 initial payload={"entities":[{"id":"stupiddevice3","type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 initial ctxt {"meas":"measIoTA","type":"StupidDevice2"} 
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 currentEntity {"id":"stupiddevice3","type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}} 
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 currentEntity sorted {"id":"stupiddevice3","type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}} 
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"id","object_id":"id","type":"text"}  newAttr="stupiddevice3"
time=2023-10-02T16:16:15.251Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="id" attr=undefined ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr=undefined 
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="id"  current entity={"type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}  current payload={"entities":[{"type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"type","object_id":"type","type":"text"}  newAttr="StupidDevice2"
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="type" attr=undefined ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr=undefined 
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="type"  current entity={"meas":{"value":"measIoTA","type":"text"}}  current payload={"entities":[{"meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="meas" attr={"name":"meas","object_id":"meas","type":"String"} ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr={"value":"measIoTA","type":"text"} 
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="meas"  current entity={"meas":{"value":"measIoTA","type":"String"}}  current payload={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 ending payload={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 payload with native types and without object_id={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-02T16:16:15.252Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=Not updating device value in the Context Broker at [http://192.168.1.1:1026/v2/op/update] due to empty payload 

[{
    "url": "http://192.168.1.1:1026/v2/op/update",
    "method": "POST",
    "headers": {
        "fiware-service": "smartgondor",
        "fiware-servicepath": "gardens"
    },
    "json": {
        "entities": [
            {
                "meas": {
                    "value": "measIoTA",
                    "type": "String"
                }
            }
        ],
        "actionType": "append"
    }
}]
AlvaroVega commented 10 months ago

Invalid mapping is detected in case 3:

time=2023-10-03T06:22:18.522Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"id","object_id":"id","type":"text"} newAttr="stupiddevice3"

mapedraza commented 10 months ago

Invalid mapping is detected in case 3:

time=2023-10-03T06:22:18.522Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"id","object_id":"id","type":"text"} newAttr="stupiddevice3"

The following log is printed at https://github.com/telefonicaid/iotagent-node-lib/tree/fix/remove_measures_with_attribute_name_id_type / https://github.com/telefonicaid/iotagent-node-lib/pull/1483/commits/cadda42d8e9ac1e1b5eb74aa8337640267dfbc88 . A request without ID and type is executed:

time=2023-10-03T06:34:56.023Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 currentEntity sorted {"id":"stupiddevice3","type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}} 
time=2023-10-03T06:34:56.023Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"id","object_id":"id","type":"text"}  newAttr="stupiddevice3"
time=2023-10-03T06:34:56.023Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="id" attr=undefined ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr=undefined 
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="id"  current entity={"type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}  current payload={"entities":[{"type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"type","object_id":"type","type":"text"}  newAttr="StupidDevice2"
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="type" attr=undefined ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr=undefined 
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="type"  current entity={"meas":{"value":"measIoTA","type":"text"}}  current payload={"entities":[{"meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="meas" attr={"name":"meas","object_id":"meas","type":"String"} ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr={"value":"measIoTA","type":"text"} 
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="meas"  current entity={"meas":{"value":"measIoTA","type":"String"}}  current payload={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 ending payload={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 payload with native types and without object_id={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=Not updating device value in the Context Broker at [http://192.168.1.1:1026/v2/op/update] due to empty payload 

[{
    "url": "http://192.168.1.1:1026/v2/op/update",
    "method": "POST",
    "headers": {
        "fiware-service": "smartgondor",
        "fiware-servicepath": "gardens"
    },
    "json": {
        "entities": [
            {
                "meas": {
                    "value": "measIoTA",
                    "type": "String"
                }
            }
        ],
        "actionType": "append"
    }
}]
AlvaroVega commented 10 months ago

LGTM

AlvaroVega commented 10 months ago

Invalid mapping is detected in case 3: time=2023-10-03T06:22:18.522Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"id","object_id":"id","type":"text"} newAttr="stupiddevice3"

The following log is printed at https://github.com/telefonicaid/iotagent-node-lib/tree/fix/remove_measures_with_attribute_name_id_type / cadda42 . A request without ID and type is executed:

time=2023-10-03T06:34:56.023Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 currentEntity sorted {"id":"stupiddevice3","type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}} 
time=2023-10-03T06:34:56.023Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"id","object_id":"id","type":"text"}  newAttr="stupiddevice3"
time=2023-10-03T06:34:56.023Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="id" attr=undefined ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr=undefined 
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="id"  current entity={"type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}  current payload={"entities":[{"type":"StupidDevice2","meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 invalid mapping for attr={"name":"type","object_id":"type","type":"text"}  newAttr="StupidDevice2"
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="type" attr=undefined ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr=undefined 
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="type"  current entity={"meas":{"value":"measIoTA","type":"text"}}  current payload={"entities":[{"meas":{"value":"measIoTA","type":"text"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 procesing j="meas" attr={"name":"meas","object_id":"meas","type":"String"} ctxt={"meas":"measIoTA","type":"StupidDevice2"} newAttr={"value":"measIoTA","type":"text"} 
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 after procesing attr="meas"  current entity={"meas":{"value":"measIoTA","type":"String"}}  current payload={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 ending payload={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=sendUpdateValueNgsi2 payload with native types and without object_id={"entities":[{"meas":{"value":"measIoTA","type":"String"}}],"actionType":"append"}
time=2023-10-03T06:34:56.024Z | lvl=DEBUG | corr=n/a | trans=n/a | op=IoTAgentNGSI.Entities-v2 | msg=Not updating device value in the Context Broker at [http://192.168.1.1:1026/v2/op/update] due to empty payload 

[{
    "url": "http://192.168.1.1:1026/v2/op/update",
    "method": "POST",
    "headers": {
        "fiware-service": "smartgondor",
        "fiware-servicepath": "gardens"
    },
    "json": {
        "entities": [
            {
                "meas": {
                    "value": "measIoTA",
                    "type": "String"
                }
            }
        ],
        "actionType": "append"
    }
}]

fixed in https://github.com/telefonicaid/iotagent-node-lib/pull/1483/commits/cadda42d8e9ac1e1b5eb74aa8337640267dfbc88