Problem with v2 subscription notification

[DiEVeXx]

I'm having some troubles with Subscription notification in Orion Context Broker.

When a notification is returned to QuantumLeap (i want to show values in Grafana), most of the time the values of the attributes arrive as strings (but sometimes not.. it's weird). So QuantumLeap gives me an exception (crate.client.exceptions.ProgrammingError: SQLActionException[ColumnValidationException: Validation failed for accelerometery: Cannot cast '0.2109375' to type long]). but all values are properly stored in Orion as Number

Here the configuration. Docker-compose files:

version: "3.3"
services:
    mongo:
        image: mongo:3.6
        ports:
            - "27017:27017"
        command: --bind_ip_all --smallfiles
        networks:
            - fiwareNetwork
        restart: on-failure

    orion:
        image: fiware/orion:2.2.0
        depends_on:
            - mongo
        expose:
            - "1026"
        ports:
            - "1026:1026"
        command: -dbhost mongo
        networks:
            - fiwareNetwork
        restart: on-failure

    quantumleap:
        image: smartsdk/quantumleap:latest
        hostname: quantumleap
        container_name: fiware-quantumleap
        ports:
          - "8668:8668"
        depends_on:
          - crate
        environment:
          - CRATE_HOST=crate
        healthcheck:
          test: curl --fail -s http://localhost:8668/v2/version || exit 1
        restart: on-failure
        networks:
          - fiwareNetwork

    crate:
        image: crate:3.3.2
        ports:
          - "4200:4200"
          - "4300:4300"
          - "5433:5432"
        command: -Ccluster.name=demoPolimi -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*" -Clicense.enterprise=false
        restart: on-failure
        networks:
          - fiwareNetwork

networks:
    fiwareNetwork:

Orion context Broker stored entity:

{
            "contextElement": {
                "type": "smartspot",
                "isPattern": "false",
                "id": "HOP840d8e003c76:smartspot",
                "attributes": [
                    {
                        "name": "accelerometerX",
                        "type": "Number",
                        "value": 0.0253906
                    },
                    {
                        "name": "accelerometerY",
                        "type": "Number",
                        "value": 0.2363281
                    },
                    {
                        "name": "accelerometerZ",
                        "type": "Number",
                        "value": 6.8183593
                    },
                    {
                        "name": "humidity",
                        "type": "Number",
                        "value": 24.3268432
                    },
                    {
                        "name": "pressure",
                        "type": "Number",
                        "value": 0
                    },
                    {
                        "name": "temperature",
                        "type": "Number",
                        "value": 36.4303817
                    },
                   ···· Some more attributes ····
                ]
            },
            "statusCode": {
                "code": "200",
                "reasonPhrase": "OK"
            }
} 

Subscription creation (I want to be notified when any attribute changes):

curl -s -X POST \
 "http://192.168.1.48:1026/v2/subscriptions" \
 -H 'Content-Type: application/json' \
 -H 'cache-control: no-cache' \
 -H 'fiware-service: smartspot' \
 -H 'fiware-servicepath: /smartspot' \
 -d "{
 \"description\":\"Notify Quantum Leap\",
 \"subject\":{
   \"entities\": [
            {
              \"type\":\"smartspot\",
              \"idPattern\":\"HOP*\"
            }
     ],
   \"condition\": {
           \"attrs\": [
           ]
   }
 },
 \"notification\": {
   \"http\": {
     \"url\":\"http://192.168.1.48:8668/v2/notify\"
   },
   \"attrs\": [],
   \"attrsFormat\": \"normalized\" 
 },
 \"throttling\":1
}"

Full error log:

fiware-quantumleap | INFO:reporter.reporter:Received payload: {'id': 'HOP840d8e003c76:smartspot', 'type': 'smartspot', 'accelerometerX': {'type': 'Number', 'value': -0.0332031, 'metadata': {}}, 'accelerometerY': {'type': 'Number', 'value': '0.1972656', 'metadata': {}}, 'accelerometerZ': {'type': 'Number', 'value': 6.78125, 'metadata': {}}, 'altitude': {'type': 'Number', 'value': 0, 'metadata': {}}, 'crowdhigherrange': {'type': 'Number', 'value': 2, 'metadata': {}}, 'crowdlowerrange': {'type': 'Number', 'value': 2, 'metadata': {}}, 'crowdmediumrange': {'type': 'Number', 'value': 2, 'metadata': {}}, 'humidity': {'type': 'Number', 'value': 24.4031372, 'metadata': {}}, 'pressure': {'type': 'Number', 'value': 0, 'metadata': {}}, 'temperature': {'type': 'Number', 'value': 36.3660316, 'metadata': {}}}
fiware-quantumleap | [2019-05-05 18:52:53,650] ERROR in app: Exception on /v2/notify [POST]
fiware-quantumleap | Traceback (most recent call last):
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
fiware-quantumleap |     response = self.full_dispatch_request()
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
fiware-quantumleap |     rv = self.handle_user_exception(e)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
fiware-quantumleap |     reraise(exc_type, exc_value, tb)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
fiware-quantumleap |     raise value
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
fiware-quantumleap |     rv = self.dispatch_request()
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
fiware-quantumleap |     return self.view_functions[rule.endpoint](**req.view_args)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 73, in wrapper
fiware-quantumleap |     response = function(request)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/uri_parsing.py", line 117, in wrapper
fiware-quantumleap |     response = function(request)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py", line 163, in wrapper
fiware-quantumleap |     response = function(request)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py", line 336, in wrapper
fiware-quantumleap |     return function(request)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 44, in wrapper
fiware-quantumleap |     response = function(request)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/connexion/decorators/parameter.py", line 207, in wrapper
fiware-quantumleap |     return function(**kwargs)
fiware-quantumleap |   File "/src/ngsi-timeseries-api/src/reporter/reporter.py", line 152, in notify
fiware-quantumleap |     trans.insert([payload], fiware_s, fiware_sp)
fiware-quantumleap |   File "/src/ngsi-timeseries-api/src/translators/crate.py", line 183, in insert
fiware-quantumleap |     fiware_servicepath)
fiware-quantumleap |   File "/src/ngsi-timeseries-api/src/translators/crate.py", line 288, in _insert_entities_of_type
fiware-quantumleap |     self.cursor.executemany(stmt, entries)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/crate/client/cursor.py", line 67, in executemany
fiware-quantumleap |     self.execute(sql, bulk_parameters=seq_of_parameters)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/crate/client/cursor.py", line 54, in execute
fiware-quantumleap |     bulk_parameters)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/crate/client/http.py", line 328, in sql
fiware-quantumleap |     content = self._json_request('POST', self.path, data=data)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/crate/client/http.py", line 448, in _json_request
fiware-quantumleap |     _raise_for_status(response)
fiware-quantumleap |   File "/usr/local/lib/python3.6/site-packages/crate/client/http.py", line 187, in _raise_for_status
fiware-quantumleap |     error_trace=error_trace)
fiware-quantumleap | crate.client.exceptions.ProgrammingError: SQLActionException[ColumnValidationException: Validation failed for accelerometery: Cannot cast '0.1972656' to type long]
fiware-quantumleap | 172.21.0.1 - - [05/May/2019 18:52:53] "POST /v2/notify HTTP/1.1" 500 -

Using an echo server docker image subscribed to the same entity I can see that some attributes of the notifications are returned as strings (accelerometerZ this time) but other times as float.

echo server notification body:

body: '{"subscriptionId":"5ccc5ca1c6d5218961ad914b","data":[{"id":"HOP840d8e003c76:smartspot","type":"smartspot","accelerometerX":{"type":"Number","value":-0.0488281,"metadata":{}},"accelerometerY":{"type":"Number","value":0.2324218,"metadata":{}},"accelerometerZ":{"type":"Number","value":"6.7265625","metadata":{}},"altitude":{"type":"Number","value":0,"metadata":{}},"crowdhigherrange":{"type":"Number","value":12,"metadata":{}},"crowdlowerrange":{"type":"Number","value":3,"metadata":{}},"crowdmediumrange":{"type":"Number","value":7,"metadata":{}},"humidity":{"type":"Number","value":22.1906127,"metadata":{}},"pressure":{"type":"Number","value":0,"metadata":{}},"temperature":{"type":"Number","value":36.5269088,"metadata":{}}}]}',

[fgalan]

It seems some kind of casting problem in the received (i.e. QuatumLeap). From the point of view of Orion, behaviour seems to be ok. I mean, attribute value is a float 0.1972656 and Orion is notifying it.

Or maybe I don't understandint the case correctly...

[fgalan]

If at the end is a problems with QuatumLeap I'd suggest to create a new issue in that repository with a reference (link to this one), then close this issue, pls. Thanks!

[DiEVeXx]

It seems some kind of casting problem in the received (i.e. QuatumLeap). From the point of view of Orion, behaviour seems to be ok. I mean, attribute value is a float 0.1972656 and Orion is notifying it.

Or maybe I don't understandint the case correctly...

If you look at the output of the echo server you can see that all attributes except the one that fails are numbers, the one that fails is being sent as a string.

I will short the entity and paste only the important things: "accelerometerX":{"value": -0.0488281} "accelerometerY":{"value": 0.2324218} "accelerometerZ":{"value": "6.7265625"}

As you can see, Orion is sending a notification with some values as Strings when the entity attribute is storing it as Number.

I think this is Orion's problem, not QuantumLeap's.

Thanks for the reply :)

[fgalan]

Could you provide the update request (verb + URL + payload) that has set these attribute values at Orion, please?

[DiEVeXx]

Could you provide the update request (verb + URL + payload) that has set these attribute values at Orion, please?

Here the logger output of the javascript code that I use to upload values to O.C.B. I'm using the javascript request library

Creating updateContext request
Request url: http://192.168.1.48:1026/v1/updateContext
Request headers: { 'fiware-service': 'smartspot',
  'fiware-servicepath': '/smartspot',
  'Content-Type': 'application/json' }
Request body: {"contextElements":[{"id":"HOP840d8e003c76:smartspot","type":"smartspot","isPattern":false,"attributes":[{"name":"accelerometerX","type":"Number","value":"0.2050781"}]}],"updateAction":"APPEND"}

TCP Stream captured using Wireshark:

POST /v1/updateContext HTTP/1.1
fiware-service: smartspot
fiware-servicepath: /smartspot
Content-Type: application/json
host: 192.168.1.48:1026
accept: application/json
content-length: 194
Connection: close

{"contextElements":[{"id":"HOP840d8e003c76:smartspot","type":"smartspot","isPattern":false,"attributes":[{"name":"accelerometerY","type":"Number","value":"0.2832031"}]}],"updateAction":"APPEND"}HTTP/1.1 200 OK
Connection: close
Content-Length: 231
Content-Type: application/json
Fiware-Correlator: a911e75a-719c-11e9-bff4-0242ac140003
Date: Wed, 08 May 2019 14:22:07 GMT

{"contextResponses":[{"contextElement":{"type":"smartspot","isPattern":"false","id":"HOP840d8e003c76:smartspot","attributes":[{"name":"accelerometerY","type":"Number","value":""}]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]}

I really don't know why GitHub is putting this answer up...

[fgalan]

There are two relevant things in the update request you are using:

1. You are using an operation of the NGSIv1 API (which is deprecated in favour of NGSIv2)
2. You are providing the the value as string: "value":"0.2832031"

A possible solution for this is:

1. To provide the value as a native JSON number, i.e 0.2832031 instead of "0.2832031"
2. To update the entity using the NGSIv2 API. There are several ways of updating in NGSIv2 (I'd recommend you to have a look to the NGSIv2 specification. This necesary, as just providing the value in native JSON number in NGSIv1 doesn't work, as NGSIv1 "stringfies" any JSON native type (numbers, booleans, etc.).

(In seems my comment is also being put up. Funny :)