Open kandarp-n opened 3 years ago
Am I supposed to raise a pull request instead (because I already found a fix)? I'm new to the open source world, so please advise :)
iotagent-node-lib/lib/services/ngsi/entities-NGSI-v1.js corresponds to an obsolete version of the API soon to be removed.
Thus, could you configure your IOTA in NGSIv2 mode and test how it goes? Or if the same problem occurs in the NGSIv2 part (I guess iotagent-node-lib/lib/services/ngsi/entities-NGSI-v2.js) it would be great if you could send it a PR with the fix.
Thanks for your feedback!
@kandarp-n it seems a typo and you has found a fix. Just edit that file in github as proposed and create a new Pull Request then.
iotagent-node-lib/lib/services/ngsi/entities-NGSI-v1.js corresponds to an obsolete version of the API soon to be removed.
Thus, could you configure your IOTA in NGSIv2 mode and test how it goes? Or if the same problem occurs in the NGSIv2 part (I guess iotagent-node-lib/lib/services/ngsi/entities-NGSI-v2.js) it would be great if you could send it a PR with the fix.
Thanks for your feedback!
@fgalan Thanks, after configuring IOTA in NGSIv2 mode, it worked fine. But when I made autoprovision=false, then too new attributes are getting created, which shouldn't get created, right?
@kandarp-n it seems a typo and you has found a fix. Just edit that file in github as proposed and create a new Pull Request then.
@AlvaroVega Thank you for confirmation. I have created a pull request.
iotagent-node-lib/lib/services/ngsi/entities-NGSI-v1.js corresponds to an obsolete version of the API soon to be removed. Thus, could you configure your IOTA in NGSIv2 mode and test how it goes? Or if the same problem occurs in the NGSIv2 part (I guess iotagent-node-lib/lib/services/ngsi/entities-NGSI-v2.js) it would be great if you could send it a PR with the fix. Thanks for your feedback!
@fgalan Thanks, after configuring IOTA in NGSIv2 mode, it worked fine. But when I made autoprovision=false, then too new attributes are getting created, which shouldn't get created, right?
I am facing similar issues. I just can't get IOTA to ignore unprovisioned attributes. Have you been able to fix it yet?
@fgalan Thanks, after configuring IOTA in NGSIv2 mode, it worked fine. But when I made autoprovision=false, then too new attributes are getting created, which shouldn't get created, right?
You are right, that shouldn't be happening... @kandarp-n maybe you could have a look and fix it the same way you fixed the other bug? :)
If confirmed, probably this is happening because we don't have proper unit tests for autoprovision=false cases. It would be great to have them.
PR #1005 has been merged. However, I understand this issue should be kept open, as some autoprovision=false cases have been notified in the latest comment.
I have tested the new image of the json-iotagent 1.17.1 which uses the lib version 2.15.1 and it worked fine for me now. -> I created a device group (/iot/services) that does not allow autoprovision. I have sent data the configured mqtt topic / apikey using an unknown deviceid. No new device has been created.
But the error message in the IoT agent looks like: time=2021-04-08T08:49:39.604Z | lvl=ERROR | corr=n/a | trans=n/a | op=IoTAgentJSON.Utils | from=n/a | srv=XXXX| subsrv=XXXX| msg=MEASURES-001: Couldn't find device data for APIKey [1234] and DeviceId[XXXX] | comp=IoTAgent The service with apikey 1234 is registered in the service "test" while the error message threw a different service (which I blurred to XXXX) that I use with other devices. -> Shouldn't this refer to the service (fiware-service, srv) that I have created the group / apikey in? -> Apparently, the log level for this is ERROR, this might spam the logs. What do you think about changing the log level to WARN or even INFO instead?
It seems that I am not able to use the explicitAttrs option .. but I'm gonna raise another issue for this.
It seems that I am not able to use the explicitAttrs option .. but I'm gonna raise another issue for this.
So this one can be closed and discussion continues in #1016?
I'm affraid not - I was too quick.
I have done a quick test again while trying something with explicitAttrs and then my first observations were ruined.
So when I tried to send data to an unprovisioned device with an already existing service
{ "commands": [], "lazy": [], "attributes": [], "_id": "6058a2c9cc3fb500a130762d", "resource": "/iot/json", "apikey": "1234", "service": "test", "subservice": "/", "autoprovision": false, "explicitAttrs": true, "__v": 0, "static_attributes": [], "internal_attributes": [], "entity_type": "Sensor:Temperature" }
no new device was created -> that what I explained in my previous post.
Then I created a new service that actually had the same settings with just the apikey changed from 1234 to 12345
{ "commands": [], "lazy": [], "attributes": [], "_id": "606ed29b5c0d990080921b32", "resource": "/iot/json", "apikey": "12345", "service": "test", "subservice": "/", "autoprovision": false, "explicitAttrs": true, "__v": 0, "static_attributes": [], "internal_attributes": [], "entity_type": "Sensor:Temperature" }
and I tried to send data to /json/12345/tsensor00X/attrs a new device was created although I set autoprovision to false.
And still the issue with the raised mongo alarm for the wrong service and the log level. Shall I create new issues for this?
I'm affraid not - I was too quick.
I have done a quick test again while trying something with explicitAttrs and then my first observations were ruined. So when I tried to send data to an unprovisioned device with an already existing service
{ "commands": [], "lazy": [], "attributes": [], "_id": "6058a2c9cc3fb500a130762d", "resource": "/iot/json", "apikey": "1234", "service": "test", "subservice": "/", "autoprovision": false, "explicitAttrs": true, "__v": 0, "static_attributes": [], "internal_attributes": [], "entity_type": "Sensor:Temperature" }
no new device was created -> that what I explained in my previous post.
So you mean this use case is working. Is my understanding correct?
Then I created a new service that actually had the same settings with just the apikey changed from 1234 to 12345
{ "commands": [], "lazy": [], "attributes": [], "_id": "606ed29b5c0d990080921b32", "resource": "/iot/json", "apikey": "12345", "service": "test", "subservice": "/", "autoprovision": false, "explicitAttrs": true, "__v": 0, "static_attributes": [], "internal_attributes": [], "entity_type": "Sensor:Temperature" }
and I tried to send data to /json/12345/tsensor00X/attrs a new device was created although I set autoprovision to false.
So you mean this use case is not working. Is my understanding correct? In that case, it would be great if you could provide the exact sequence of steps to reproduce the problem (curl and mqtt_pulblish statements are preferred) since you start the agent with empty database.
And still the issue with the raised mongo alarm for the wrong service and the log level. Shall I create new issues for this?
No issue needed. A PR solving it is on the way: https://github.com/telefonicaid/iotagent-node-lib/pull/1017
I'm affraid not - I was too quick.
I have done a quick test again while trying something with explicitAttrs and then my first observations were ruined. So when I tried to send data to an unprovisioned device with an already existing service { "commands": [], "lazy": [], "attributes": [], "_id": "6058a2c9cc3fb500a130762d", "resource": "/iot/json", "apikey": "1234", "service": "test", "subservice": "/", "autoprovision": false, "explicitAttrs": true, "__v": 0, "static_attributes": [], "internal_attributes": [], "entity_type": "Sensor:Temperature" } no new device was created -> that what I explained in my previous post.
So you mean this use case is working. Is my understanding correct?
I have no idea what happened there and why the autoprovision=false worked in this case. So I am just gonna respond to your request
So you mean this use case is not working. Is my understanding correct? In that case, it would be great if you could provide the exact sequence of steps to reproduce the problem (curl and mqtt_pulblish statements are preferred) since you start the agent with empty database.
I have set up a fresh setup via docker-compose using fiware/orion:2.5.2, fiware/iotagent-json:1.17.1, mongo:4.4 and eclipse-mosquitto:2.0.10.
I created a service group
curl -iX POST \ 'http://localhost:4041/iot/services' \ -H 'Content-Type: application/json' \ -H 'fiware-service: test' \ -H 'fiware-servicepath: /' \ -d '{ "services": [ { "apikey": "test", "cbroker": "http://orion:1026", "entity_type": "Sensor:Temperature", "resource": "/iot/json", "autoprovision": false, "explicitAttrs": true } ] }'
Created a device
curl -iX POST \ 'http://localhost:4041/iot/devices' \ -H 'Content-Type: application/json' \ -H 'fiware-service: test' \ -H 'fiware-servicepath: /' \ -d '{ "devices": [ { "device_id": "tsensor001", "entity_name": "urn:ngsi-ld:Sensor:Temperature:001", "entity_type": "Sensor:Temperature", "transport": "MQTT", "attributes": [ { "object_id": "t", "name": "presentValue", "type": "Number" } ], "lazy": [], "commands": [], "static_attributes": [], "internal_attributes": [], "protocol": "PDI-IoTA-JSON", "explicitAttrs": true }] }'
I wasn't able to use the docker run mosquitto-pub way that has been used in the step-by-step tutorials but used some other client instead to publish messages to mosquitto instead.
After re-reading the issue I realized I was using the environment variable IOTA_CB_NGSI_VERSION=v2 so when I changed it to v1 it worked almost as expected with the same procedure as described above: Command 1) worked fine, with command 2) no new attribute was created but with command 3) a new entitiy was still created but it only contained the Timestamp and no other attribute; so it ignored the one that I actually sent.
So is this an issue of using the correct ngsi-lib version -> there is no autocast / explicit attrs part in the code of the libs despite the v1 one that has been updated lately by @kandarp-n .
And still the issue with the raised mongo alarm for the wrong service and the log level. Shall I create new issues for this?
No issue needed. A PR solving it is on the way: #1017
PR merged. The problem with false positive in mongo alarm should now be solved in master branch. It would be great if you could provide feedback about it :)
I exec a manual functionality using iotagent-json in order to define what is happening now:
curl -iX POST "http://localhost:7896/iot/json?k=testapikey&i=dev1" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
curl -iX POST "http://localhost:7896/iot/json?k=testapikey&i=dev4" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Defining autoprovision=true
in cofig group:
Defining autoprovision=false
in config group:
Additional info: IOTA_APPEND_MODE
not defined
Tests to be done also in IOTA_APPEND_MODE=true
case
The results of the previous tests using the ENV_VAR IOTA_APPEND_MODE=true
are exactly the same that
As additional information, the tests in charge of validate are available here
After merging PR #1048 this issue could have been solved.
@SBlechmann it would be great if you could test it using :latest
tag from IOTAs from dockerhub.
@SBlechmann it would be great if you could test it using :latest tag from IOTAs from dockerhub.
In particular, these ones:
telefonicaiot/iotagent-ul:latest
telefonicaiot/iotagent-json:latest
After merging #1048 the tests were performed once again using telefonicaiot/iotagent-json:latest
with the following result
The docker-compose file used to perform the checks are here
According to the description here, the IoTA should perform an update (avoiding autoprovision)
We create a device group with autoprovision
=false
curl -iX POST "http://localhost:4041/iot/services" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"services": [
{
"apikey": "apikeytest",
"cbroker": "http://orion:1026",
"entity_type": "Sensor:Temperature",
"resource": "/iot/json",
"autoprovision": false
}
]
}'
Response
{}
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest&i=dev1" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
curl --location --request GET 'localhost:1026/v2/entities' \
--header 'Fiware-Service: tests' \
--header 'Fiware-ServicePath: /'
Response
[{"id":"Sensor:Temperature:dev1","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:15:17.705Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:15:17.705Z"}}},"t":{"type":"string","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:15:17.705Z"}}}}]
This is a NOK result, the device was created with 2 atributes sent from IoT Agent.
Then, we can delete the entity wit curl -iX DELETE "http://localhost:1026/v2/entities/Sensor:Temperature:dev1" -H 'fiware-service: tests' -H 'fiware-servicepath: /'
curl -iX POST "http://localhost:4041/iot/devices" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"devices": [
{
"device_id": "dev2",
"entity_name": "Sensor:Temperature:dev2",
"entity_type": "Sensor:Temperature",
"attributes": [
{
"object_id": "t",
"name": "temperature",
"type": "float"
}
],
"transport": "HTTP"
}
]
}'
If we check the entity (GET /v2/entities), we can se an entity created with only one atribute, temperature
.
[{"id":"Sensor:Temperature:dev2","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:26:09.284Z","metadata":{}},"temperature":{"type":"float","value":null,"metadata":{}}}]V
In this case, we are going to post 2 parameters through IoTAgent to check if is adding or not a new atributte
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest&i=dev2" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Since the provision was not defining the autoprovision
value, it is expected a new parameter on the Entity. Getting the entity from CB:
[{"id":"Sensor:Temperature:dev2","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:31:09.912Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:31:09.912Z"}}},"temperature":{"type":"float","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:31:09.912Z"}}}}]
This is a NOK result, since the h parameter was created
Then, we can delete the entity wit curl -iX DELETE "http://localhost:1026/v2/entities/Sensor:Temperature:dev2" -H 'fiware-service: tests' -H 'fiware-servicepath: /'
According to the description here, the IoTA should perform an update (avoiding autoprovision)
curl -iX POST "http://localhost:4041/iot/devices" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"devices": [
{
"device_id": "dev3",
"entity_name": "Sensor:Temperature:dev3",
"entity_type": "Sensor:Temperature",
"attributes": [
{
"object_id": "t",
"name": "temperature",
"type": "float"
}
],
"transport": "HTTP",
"autoprovision": false
}
]
}'
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest&i=dev3" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Since the provision was defining the autoprovision
false, it is not expected a new attribute on the Entity. Getting the entity from CB:
[{"id":"Sensor:Temperature:dev3","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:39:11.266Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:39:11.266Z"}}},"temperature":{"type":"float","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:39:11.266Z"}}}}]
In this case, a new attribute h
was created. This is a NOK
Then, we can delete the entity wit curl -iX DELETE "http://localhost:1026/v2/entities/Sensor:Temperature:dev3" -H 'fiware-service: tests' -H 'fiware-servicepath: /'
According to the description here, the IoTA should perform an append reques (alowing autoprovision)
curl -iX POST "http://localhost:4041/iot/devices" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"devices": [
{
"device_id": "dev4",
"entity_name": "Sensor:Temperature:dev4",
"entity_type": "Sensor:Temperature",
"attributes": [
{
"object_id": "t",
"name": "temperature",
"type": "float"
}
],
"transport": "HTTP",
"autoprovision": true
}
]
}'
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest&i=dev4" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Since the provision was defining the autoprovision
true, it is expected a new attribute on the Entity. Getting the entity from CB:
[{"id":"Sensor:Temperature:dev4","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:44:05.736Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:44:05.736Z"}}},"temperature":{"type":"float","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:44:05.736Z"}}}}]
In this case, a new attribute h
was created. This is a OK
Then, we can delete the entity wit curl -iX DELETE "http://localhost:1026/v2/entities/Sensor:Temperature:dev4" -H 'fiware-service: tests' -H 'fiware-servicepath: /'
curl -iX POST "http://localhost:4041/iot/services" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"services": [
{
"apikey": "apikeytest2",
"cbroker": "http://orion:1026",
"entity_type": "Sensor:Temperature",
"resource": "/iot/json",
"autoprovision": true
}
]
}'
Post data
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest2&i=dev5" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Device Created
[{"id":"Sensor:Temperature:dev5","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:53:51.300Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:53:51.300Z"}}},"t":{"type":"string","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:53:51.300Z"}}}}]
This is a OK
Create device
curl -iX POST "http://localhost:4041/iot/devices" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"devices": [
{
"device_id": "dev6",
"entity_name": "Sensor:Temperature:dev6",
"entity_type": "Sensor:Temperature",
"attributes": [
{
"object_id": "t",
"name": "temperature",
"type": "float"
}
],
"transport": "HTTP",
"autoprovision": false
}
]
}'
Post data
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest2&i=dev6" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Device Created
[{"id":"Sensor:Temperature:dev6","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:52:11.071Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:52:11.071Z"}}},"temperature":{"type":"float","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:52:11.071Z"}}}}]
This is a NOK
Create device
curl -iX POST "http://localhost:4041/iot/devices" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"devices": [
{
"device_id": "dev7",
"entity_name": "Sensor:Temperature:dev7",
"entity_type": "Sensor:Temperature",
"attributes": [
{
"object_id": "t",
"name": "temperature",
"type": "float"
}
],
"transport": "HTTP",
"autoprovision": true
}
]
}'
Post data
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest2&i=dev7" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Device Created
[{"id":"Sensor:Temperature:dev7","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T09:58:05.467Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:58:05.467Z"}}},"temperature":{"type":"float","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T09:58:05.467Z"}}}}]
This is a OK
Create device
curl -iX POST "http://localhost:4041/iot/devices" -H 'Content-Type: application/json' -H 'fiware-service: tests' -H 'fiware-servicepath:/' -d '{
"devices": [
{
"device_id": "dev8",
"entity_name": "Sensor:Temperature:dev8",
"entity_type": "Sensor:Temperature",
"attributes": [
{
"object_id": "t",
"name": "temperature",
"type": "float"
}
],
"transport": "HTTP"
}
]
}'
Post data
curl -iX POST "http://localhost:7896/iot/json?k=apikeytest2&i=dev8" -H 'Content-Type: application/json' -d '{"h":98,"t":99}'
Device Created
[{"id":"Sensor:Temperature:dev8","type":"Sensor:Temperature","TimeInstant":{"type":"DateTime","value":"2021-05-26T10:00:19.987Z","metadata":{}},"h":{"type":"string","value":98,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T10:00:19.987Z"}}},"temperature":{"type":"float","value":99,"metadata":{"TimeInstant":{"type":"DateTime","value":"2021-05-26T10:00:19.987Z"}}}}]
This is a OK
The behavior is mostly the same as before, without any change using autoprovision
parameter
With autoprovision
= false
on the device group:
autoprovision
parameter: If message send, the attributes are included - NOKautoprovision
= false
: If message send, the attributes are included - NOKautoprovision
= true
: If message send, the attributes are included - OKWith autoprovision
= true
on the device group:
autoprovision
parameter: If message send, the attributes are included - OKautoprovision
= false
: If message send, the attributes are included - NOKautoprovision
= true
: If message send, the attributes are included - OKIt seems that autoprovision is used for ngsiv1: https://github.com/telefonicaid/iotagent-node-lib/blob/3bbc3c2c817d5f9913b2f74cc90b3b3dfd4cc955/lib/services/ngsi/entities-NGSI-v1.js#L243-L253 but not for ngsiv2 https://github.com/telefonicaid/iotagent-node-lib/blob/master/lib/services/ngsi/entities-NGSI-v2.js
So it is not enough be properly configured, should be also used (still not implemented in ngsiv2 ?)
PR with the fix for NGSIv2: https://github.com/telefonicaid/iotagent-node-lib/pull/1051
Issue: Autoprovision functionality not working as expected when configured at device/group level.
Steps to reproduce: 1) Register a device with 'autoprovision' flag set to true:
2) Send measurement for this device with an extra measurement attribute. Sample measurement payload:
Expected result: As "autoprovision" is "true" for this device, the new attribute "humidity" should get created in the Orion entity with its measurement value. Actual result: "humidity" attibute is not getting created in Orion entity. HTTP status code 404 is returned. (Note: Measurement of "temperature" is getting updated in Orion though.)
I've identified potential cause of the issue: https://github.com/telefonicaid/iotagent-node-lib/blob/8a6cbd1a8d4023f239195f1aa1534c75a487f139/lib/services/ngsi/entities-NGSI-v1.js#L243-L249
Below fix is required (line# 246)