orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

Array parameters #24

Closed netzahdzc closed 6 years ago

netzahdzc commented 7 years ago

It seems that QuantumLeap is struggling when catching Array data. I have noticed that the Orion is receiving the data but never store as historical (on the Quantum side).

This is an example of the json sent:

{
 "id": "smartphone-9845C",
 "type": "Device",
 "category": {
    "value": "smartphone"
    },
 "osVersion": {
    "value": "Android 4.0"
    },
 "softwareVersion": {
    "value": "MA-Test 1.6"
    },
 "hardwareVersion": {
    "value": "GP-P9872"
    },
 "firmwareVersion": {
    "value": "SM-A310F"
    },
 "consistOf":{
    "value": 
        [
         "sensor-9845A",
         "sensor-9845B",
         "sensor-9845C"
        ]
    },
 "refDeviceModel": {
    "value": "myDevice-345"
    },
 "dateCreated": {
    "value": "2016-08-22T10:18:16Z"
    }
}

This is the error I found when checking the Quantum's Log: crate.client.exceptions.ProgrammingError: SQLActionException[ColumnValidationException: Validation failed for refdevice: ['device-9845A', 'device-9845B', 'device-9845C'] cannot be cast to type object]

taliaga commented 7 years ago

@netzahdzc, thanks for providing the details! As promised, I'm looking at the problem here.

As explained in the docs, you need to indicate the attributes types (for now). You might be able to work around this issue by only adding the "type": "Array" to the "consistOf" attribute as shown below. The rest might not be necessary because the defaults are already well aligned with QL.

...
"consistOf":{
    "type": "Array", 
    "value": 
        [
         "sensor-9845A",
         "sensor-9845B",
         "sensor-9845C"
        ]
},
...

Complementing, I'll try to better understand what orion is doing with the default datatypes for arrays and dicts. This will give a better idea of which are the datatypes Quantum should be prepared to handle. Note QL is already accepting arrays of strings and dicts (structuredValues), but I need to align this with Orion's defaults so that we can later skip passing the "type" for the attributes as a whole.

Let me know if that change lets you move forward in the meantime.

netzahdzc commented 7 years ago

ok, thanks!!

taliaga commented 7 years ago

Just an update after inspecting orion's behavior, the problem is Orion will apply the same type for both arrays and objects. Both will be tagged as StructuredValue :(. See this PR in Orion for more details.

QL was prepared to treat StructuredValue as objects, not as arrays. In your example, we're talking about attribute consistOf. That's the root of the problem.

The task to do now in QL to fix this issue is to inspect the value and deduce/assign the corresponding type. The fix for this issue should be aligned with the resolution of #36.

taliaga commented 6 years ago

Fixed in https://github.com/smartsdk/ngsi-timeseries-api/pull/53/files.

Feel free to reopen if this problem persists