Non-Primitive Values from a Registered Content Provider are not showing a value

[jason-fox]

Orion /version:

{
    "orion": {
        "version": "1.13.0-next",
        "uptime": "0 d, 16 h, 51 m, 32 s",
        "git_hash": "41276b3260b17fa2e20fc2be58767c786df9e984",
        "compile_time": "Tue Apr 17 17:08:42 UTC 2018",
        "compiled_by": "root",
        "compiled_in": "afaf7954cd8f",
        "release_date": "Tue Apr 17 17:08:42 UTC 2018",
        "doc": "https://fiware-orion.readthedocs.org/en/master/"
    }
}

I'm using the /v2/registrations endpoint to register a content provider with the legacyForwarding flag being set. Therefore my Content Provider is offering the v1/queryContext endpoint

When I am returning a simple value (Integer, String etc.) such as a temperature the data is added to the context correctly:

{
    "contextResponses": [
        {
            "contextElement": {
                "attributes": [
                    {
                        "name": "temperature",
                        "type": "Number",
                        "value": 27
                    }
                ],
                "id": "urn:ngsi-ld:Store:001",
                "isPattern": "false",
                "type": "Store"
            },
            "statusCode": {
                "code": "200",
                "reasonPhrase": "OK"
            }
        }
    ]
}

However when trying to return an array of strings as shown from a Context Provider.

{
    "contextResponses": [
        {
            "contextElement": {
                "attributes": [
                    {
                        "name": "tweets",
                        "type": "Array",
                        "value": [
                            "String 1",
                            "String 2"
                        ]
                    }
                ],
                "id": "urn:ngsi-ld:Store:002",
                "isPattern": "false",
                "type": "Store"
            },
            "statusCode": {
                "code": "200",
                "reasonPhrase": "OK"
            }
        }
    ]
}

I can see the request being sent in the log and I can retrieve the following entity:

{
    "id": "urn:ngsi-ld:Store:002",
    "type": "Store",
    "address": {
        "type": "PostalAddress",
        "value": "",
        "metadata": {}
    },
    "location": {
        "type": "geo:json",
        "value": "",
        "metadata": {}
    },
    "name": {
        "type": "Text",
        "value": "Checkpoint Markt",
        "metadata": {}
    },
    "tweets": {
        "type": "Array",
        "value": "",
        "metadata": {}
    }
}

As you can see the "tweets" value is blank, but the attribute exists and the type has been successfully received.

On further investigation I see that even simple values are causing a problem in that if I have an entity with an Object

e.g.:

{
    "id": "urn:ngsi-ld:Store:003",
    "type": "Store",
    "address": {
        "type": "PostalAddress",
        "value": {
            "streetAddress": "Mühlenstrasse 10",
            "addressRegion": "Berlin",
            "addressLocality": "Friedrichshain",
            "postalCode": "10243"
        },
        "metadata": {}
    }
}

and I register a simple attribute such as temperature, I get back the following:

{
    "id": "urn:ngsi-ld:Store:003",
    "type": "Store",
    "address": {
        "type": "PostalAddress",
        "value": "",
        "metadata": {}
    },
   "temperature": {
        "type": "Number",
        "value": "14.3",
        "metadata": {}
    }
}

As you can see the Object is no longer visible - furthermore a direct request to the attribute:

http://{{orion}}/v2/entities/urn:ngsi-ld:Store:003/attrs/address

Returns a blank "" as shown:

{
    "type": "PostalAddress",
    "value": "",
    "metadata": {}
}

The Issue was originally raised on Stack Overflow

[fgalan]

Associated functional tests in PR https://github.com/telefonicaid/fiware-orion/pull/3163

[fgalan]

Thanks for the report!

The behaviour you describe seems to be a bug in the forwarding logic with regards to arrays and objects. The issue is going to be labeled so.

[fgalan]

Description of the current behaviour included in documentation in PR https://github.com/telefonicaid/fiware-orion/pull/3210

[fgalan]

Once this issue gets fixed and before closing it, remember to remove the following paragraph in doc/manuals/user/context_providers.md:

At the present moment, Context Broker is not able to include compound attributes in forwarded updates. A blank ("") if forwarded for them instead. Please have a look to the following issue for more information.

[versatildefuy]

Hi, we are trying to represent the Fiware DataModels and we are stuck with this issue. Is there any resolution forecast?

[mainakae]

We are working in some projects in need of this functionality (integrating with other fiware components) and are waiting too for it to be solved :)

Thx!

[fgalan]

@versatildefuy @mainakae could you elaboraton on how the Orion instance you are using is deployed, please? I mean:

• On premise deployment based on docker at dockerhub
• On premise deployment based in RPM
• On premise deployment based in copiled sources
• Orion instance running in third party (which one?) cloud
• other

Thx!

[mainakae]

Sure, we are using a dockerized on premises installation, and we have set up an integration with an IoT Agent to serve as intermediary between mqtt and Orion.

The IoT Agent registers itself as CP, and the exchange of information via JSON objects in attributes is thus inhibited.

[fgalan]

The situation is as follows:

We currently have a patch that should fix this problem. However, the patch could be introducing a memory leak. We are not fully sure, but investigating if the leak is there or not is hard and can take time. The outcome of that investigation is precluding to introduce that patch in the master branch, as master branch has to be as leak free as possible and we cannot take the risk of introduce leaks.

However, we can do the following:

• Create a new branch based on today master (i.e. bugfixing/3162_fix_but_potential_leak)
• Apply the patch to that branch
• Create a docker label based in that branch (e.g. "fiware/orion:fix3162") in dockerhub
• You can deploy that docker and benefit from the fix

Eventually a leak-fix will be developed for master branch but on the meanwhile at least you would have a Orion docker that should solve your case.

@mainakae what do you think? @versatildefuy that would also solve your case?

[mainakae]

I would really like it! we are in a development phase, more interested in getting a PoC or a prototype working with full functionality, than being absolutely production ready. We can provide whatever feedback we can gather about the working of Orion in the mean time, if you like.

Thanks!

[versatildefuy]

Yes, that would serve also for us. We can try the fix and report if any bug is encountered. Thank you very much @fgalan for your quick response.

[fgalan]

Docker firare/orion:fix3162 ready to deploy and use.

Please @versatildefuy and @mainakae tell us how it goes posting comments in this issue. Thx!

[mainakae]

Hi @fgalan I've just tried the new setup and It's working nicely with API v1 ¡great!.

I've raised my expectations a little bit too much, I'm afraid, and as I've seen that API v2 is being adopted in iotagent-node-lib and iotagent-json, I've tried the same test but issuing commands with API v2 and it didn't work.

As of right now, it seems that the context provider update with non-primitives is working only in v1 API.

I'll report any issues that I might encounter while working with orion that may lead to further debugging the patch.

Thanks again! Pedro.

[fgalan]

At the present moment, due to lack of definition of NGSIv2-based forwarding, CB always uses NGSIv1 in the CB->IOTAgent forwarding. From a client perspective, there is no change in the forwarding no matter if client uses NGSIv1 or NGSIv2 to do the update at CB endpoint.

Thus, maybe the problem you have with NGSIv2 is caused by another cause. Could you add which exact request are you using to update CB in both NGSIv1 (working fine) and NGSIv2 (failing), please?

[mainakae]

Sure, here they are:

### Send command to TheDevice1 with NGSIv1 - Envio de un ON
POST http://localhost:1026/v1/updateContext
Accept: application/json
Fiware-Service: riego
Fiware-ServicePath: /pamplona
Content-Type: application/json

{
  "contextElements": [{
    "id": "riego_01",
    "type": "Riego",
    "isPattern": "false",
    "attributes": [{
      "name": "valve",
      "type": "command",
      "value": {
        "num": "1"
      }
    }]
  }],
  "updateAction": "UPDATE"
}

### Send command to TheDevice1 API v2
PATCH http://localhost:1026/v2/entities/riego_01/attrs
Accept: application/json
Fiware-Service: riego
Fiware-ServicePath: /pamplona
Content-Type: application/json

{
  "valve": {
    "value": {"num": 12},
    "type": "command"
  }
}

[fgalan]

According to https://fiware-orion.readthedocs.io/en/master/user/context_providers/index.html

On forwarding, any type of entity in the NGSIv2 update/query matches registrations without entity type. However, the opposite doesn't work, so if you have registrations with types, then you must use ?type in NGSIv2 update/query in order to obtain a match. Otherwise you may encounter problems, like the one described in this post at StackOverflow.

or to https://fiware-iot-stack.readthedocs.io/en/master/device_api/index.html#send-commands

It is important to note that parameter type, with the entity_type must be included.

could you test with

PATCH http://localhost:1026/v2/entities/riego_01/attrs?type=Riego

and tell us how it goes?

[mainakae]

Off course, you are absolutely right, It solved the problem completely.

Sorry (and ashamed) not to have properly rtfm XD

Thanks again @fgalan you where most helpful and kind. Pedro.

[versatildefuy]

In our case, we are having the same issue. I'll try to explain better our case. Currently, we are developing a kind of IoT Agent based in our devices requirements and we are trying to represent the FIWARE Data Models. We have separated the attributes in an IoT Agent similar way so at the end we are trying to register an entity with four lazy attributes and some other statics. The lazy attributes are registered in Orion with v1 API as can be seen in the following request:

Query  http://{{url}}:{{port-orion}}/v1/registry/discoverContextAvailability
Response
{
    "contextRegistrationResponses": [
        {
            "contextRegistration": {
                "entities": [
                    {
                        "type": "Device",
                        "isPattern": "false",
                        "id": "hop-ubiquitous:Device"
                    }
                ],
                "attributes": [
                    {
                        "name": "category",
                        "type": "ListOfText",
                        "isDomain": "false"
                    },
                    {
                        "name": "mnc",
                        "type": "Text",
                        "isDomain": "false"
                    },
                    {
                        "name": "mcc",
                        "type": "Text",
                        "isDomain": "false"
                    },
                    {
                        "name": "ipAddress",
                        "type": "Text",
                        "isDomain": "false"
                    }
                ],
                "providingApplication": "http://192.168.1.192:8080/v1"
            }
        }
    ]
}

So, when we try to recover the values of this entity with the different methods and versions provided by Orion we receive the following responses:

Query http://{{url}}:{{port-orion}}/v1/contextEntities
Response
{
    "contextResponses": [
        {
            "contextElement": {
                "type": "Device",
                "isPattern": "false",
                "id": "hop-ubiquitous:Device",
                "attributes": [
                    {
                        "name": "controlledAsset",
                        "type": "ListOfText",
                        "value": [
                            "DeviceModel",
                            "AirQualityObserved"
                        ]
                    },
                    {
                        "name": "controlledProperty",
                        "type": "ListOfText",
                        "value": [
                            "temperature",
                            "humidity"
                        ]
                    },
                    {
                        "name": "macAddress",
                        "type": "ListOfText",
                        "value": [
                            "00:00:00:00:00:00"
                        ]
                    },
                    {
                        "name": "supportedProtocol",
                        "type": "Text",
                        "value": "OMA LWM2M"
                    },
                    {
                        "name": "category",
                        "type": "ListOfText",
                        "value": [
                            "Smart Cities"
                        ]
                    },
                    {
                        "name": "mnc",
                        "type": "Text",
                        "value": "9"
                    },
                    {
                        "name": "mcc",
                        "type": "Text",
                        "value": "10"
                    },
                    {
                        "name": "ipAddress",
                        "type": "Text",
                        "value": "192.168.1.1"
                    }
                ]
            },
            "statusCode": {
                "code": "200",
                "reasonPhrase": "OK"
            }
        }
    ]
}

Query http://{{url}}:{{port-orion}}/v1/contextEntities?attributeFormat=object
Response
{
    "contextResponses": [
        {
            "contextElement": {
                "type": "Device",
                "isPattern": "false",
                "id": "hop-ubiquitous:Device",
                "attributes": {
                    "controlledAsset": {
                        "type": "ListOfText",
                        "value": [
                            "DeviceModel",
                            "AirQualityObserved"
                        ]
                    },
                    "controlledProperty": {
                        "type": "ListOfText",
                        "value": [
                            "temperature",
                            "humidity"
                        ]
                    },
                    "macAddress": {
                        "type": "ListOfText",
                        "value": [
                            "00:00:00:00:00:00"
                        ]
                    },
                    "supportedProtocol": {
                        "type": "Text",
                        "value": "OMA LWM2M"
                    },
                    "category": {
                        "type": "ListOfText",
                        "value": [
                            "Smart Cities"
                        ]
                    },
                    "mnc": {
                        "type": "Text",
                        "value": "9"
                    },
                    "mcc": {
                        "type": "Text",
                        "value": "10"
                    },
                    "ipAddress": {
                        "type": "Text",
                        "value": "192.168.1.1"
                    }
                }
            },
            "statusCode": {
                "code": "200",
                "reasonPhrase": "OK"
            }
        }
    ]
}

Query http://{{url}}:{{port-orion}}/v2/entities?limit=50
Response
[
    {
        "id": "hop-ubiquitous:Device",
        "type": "Device",
        "controlledAsset": {
            "type": "ListOfText",
            "metadata": {}
        },
        "controlledProperty": {
            "type": "ListOfText",
            "metadata": {}
        },
        "macAddress": {
            "type": "ListOfText",
            "metadata": {}
        },
        "supportedProtocol": {
            "type": "Text",
            "value": "OMA LWM2M",
            "metadata": {}
        },
        "category": {
            "type": "ListOfText",
            "value": "",
            "metadata": {}
        },
        "mnc": {
            "type": "Text",
            "value": "9",
            "metadata": {}
        },
        "mcc": {
            "type": "Text",
            "value": "10",
            "metadata": {}
        },
        "ipAddress": {
            "type": "Text",
            "value": "192.168.1.1",
            "metadata": {}
        }
    }
]

Query http://{{url}}:{{port-orion}}/v2/entities?options=keyValues,count&limit=50
Response
[{"id":"hop-ubiquitous:Device","type":"Device","controlledAsset":,"controlledProperty":,"macAddress":,"supportedProtocol":"OMA LWM2M","category":"","mnc":"9","mcc":"10","ipAddress":"192.168.1.1"}]

Concluding, with this bug we are unable to represent any data model with actual device data.

Please @fgalan correct me if I'm wrong or I have any mistake in some step. Thank you in advance

[fgalan]

@versatildefuy not sure if I'm understanding the problem you describe...

Of the 4 queries you show, which ones would be right and which ones would be wrong? (citing URL in the response to this question is enough, thanks)

[versatildefuy]

Those are correct: http://{{url}}:{{port-orion}}/v1/contextEntities http://{{url}}:{{port-orion}}/v1/contextEntities?attributeFormat=object and those are failing: http://{{url}}:{{port-orion}}/v2/entities?limit=50 http://{{url}}:{{port-orion}}/v2/entities?options=keyValues,count&limit=50

In v1 requests the complex data structures are represented and in the v2 are substituted with empty strings ("")

[fgalan]

It is a known issue that pagination and forwarding don't work good together. Details can be found in the following issues: https://github.com/telefonicaid/fiware-orion/issues/847#issuecomment-107379768, https://github.com/telefonicaid/fiware-orion/issues/1647 and https://github.com/telefonicaid/fiware-orion/issues/3263. I understand that the case you are describing is related with one (or more) of them.

Leaving this appart (that should be discussed in the above issues) and focusing in this issue (#3162), I understand the first two queries (i.e. the righ ones) add to the positive report from @mainakae and confirm that the fiware/orion:fix3162 docker fixes the issue.

However, the issue will keep opened while a definitive and leak-proof solution will be developed for master branch.

[versatildefuy]

Ok, thank you very much for your diagnosis. I retried the v2 request tests without pagination options: http://{{url}}:{{port-orion}}/v2/entities http://{{url}}:{{port-orion}}/v2/entities?options=keyValues and the problem is still the same: empty string in structured values, arrays in our case. I think we can conclude is not a pagination problem. Am I wrong?

So, in order to treat this properly, how do you want to proceed? Do you prefer a new issue concerning this? Are there other issues related to this problem? Thank you again for your work

[fgalan]

Maybe it is related with I said at https://github.com/telefonicaid/fiware-orion/issues/3162#issuecomment-422360999

Could you try including entity type in the URL, pls? I mean something like:

http://{{url}}:{{port-orion}}/v2/entities?type=<entity_type>
http://{{url}}:{{port-orion}}/v2/entities?options=keyValues&type=<entity_type>

[versatildefuy]

It seems it is not related. I tried both requests you proposed with the same result: empty strings. Any other idea or test I could perform? Thank you again

[fgalan]

Could you confirm which Orion version are you running? I mean, the result of GET /version. Thx!

[versatildefuy]

I'm using fiware/orion:fix3162 docker image. The one you create for this issue. Here is the response to GET /version request:

{
    "orion": {
        "version": "1.15.0-next",
        "uptime": "0 d, 0 h, 7 m, 9 s",
        "git_hash": "71eb582d669b344e40961c0c99fa35def9df2800",
        "compile_time": "Fri Sep 14 12:24:06 UTC 2018",
        "compiled_by": "root",
        "compiled_in": "7496ef0e0c91",
        "release_date": "Fri Sep 14 12:24:06 UTC 2018",
        "doc": "https://fiware-orion.readthedocs.org/en/master/"
    }
}

Thanks

[fgalan]

Note that the HEAD commit in branch bugfixing/3162_fix_but_potential_leak is 231d2f4

However, the git_hash field in your version output shows 71eb582 which is not part of the branch, but of master:

Thus, it seems you are not testing with the right Orion version...

[fgalan]

To discard docker fiware/orion:fix3162 was built incorrectly, @mainakae could you post the result of GET /version in your case, pls?

[dherykw]

I have building the same docker image

mongo:
  image: mongo:3.4
  ports:
    - "27017:27017"
  command: --nojournal
orion:
  image: fiware/orion:fix3162
  links:
    - mongo
  ports:
    - "1026:1026"
  command: -dbhost mongo -logForHumans -logLevel DEBUG

and I am retrieving the same Orion version than @versatildefuy

{
"orion" : {
  "version" : "1.15.0-next",
  "uptime" : "0 d, 0 h, 1 m, 1 s",
  "git_hash" : "71eb582d669b344e40961c0c99fa35def9df2800",
  "compile_time" : "Fri Sep 14 12:24:06 UTC 2018",
  "compiled_by" : "root",
  "compiled_in" : "7496ef0e0c91",
  "release_date" : "Fri Sep 14 12:24:06 UTC 2018",
  "doc" : "https://fiware-orion.readthedocs.org/en/master/"
}
}

Live Long and Prosper plus data model support for IoT agents

[fgalan]

@versatildefuy @mainakae please re-deploy new image for fiware/orion:fix3162. I missed some changes in the branch (now done at https://github.com/telefonicaid/fiware-orion/commit/e77449239487395299e15f8e557cb657db481768) that were plecluding of building the righ code image.

The right commit you should see now in git_hash field in GET /version is e77449239487395299e15f8e557cb657db481768.

The weird thing is that @mainakae provided a positive report (see https://github.com/telefonicaid/fiware-orion/issues/3162#issuecomment-422366233) in a version taken from master branch (i.e. without the modification aimed to solve the bug). That makes me have doubts... do we have a real bug? :/

[versatildefuy]

Good news @fgalan !!!

The issue is resolved in the new docker image. I made a small test just now and it seems everything is working properly. I'll continue with more tests next week, providing reports if any problem is detected.

Regarding the presence or absence of bugs, I can definitely confirm you there was a bug with v2 forwarding over structured values.

Thank you very much for your responses

[fgalan]

I have been looking into this a little bit...

It seems that forwarding compound values works in the master branch right now. That would explain the positive @mainakae at https://github.com/telefonicaid/fiware-orion/issues/3162#issuecomment-422366233 even when the docker used by him was one from master, as he was using updates.

Thus, the scope of the bug is forwarded queries only. That would explain the result of @versatildefuy tests (not working in master but working in fiware/orion:fix3162 image), as he was using queries.

Is my interpretation correct?

In positive case, the documentation added before (see https://github.com/telefonicaid/fiware-orion/issues/3162#issuecomment-396646791)

At the present moment, Context Broker is not able to include compound attributes in forwarded updates. A blank ("") if forwarded for them instead. Please have a look to the following issue for more information.

is incorrect and should be changed to

At the present moment, Context Broker is not able to include compound attributes in results of forwared queries. A blank ("") if forwarded for them instead. Please have a look to the following issue for more information. However, note that updates includinig compound values doesn't have this problem and are forwarded correctly.

[fgalan]

PR with the doc fix: https://github.com/telefonicaid/fiware-orion/pull/3320

[mainakae]

To discard docker fiware/orion:fix3162 was built incorrectly, @mainakae could you post the result of GET /version in your case, pls?

Sorry for the late reply, here is the response to the GET /version

{
"orion" : {
  "version" : "1.15.0-next",
  "uptime" : "0 d, 0 h, 1 m, 40 s",
  "git_hash" : "71eb582d669b344e40961c0c99fa35def9df2800",
  "compile_time" : "Fri Sep 14 12:24:06 UTC 2018",
  "compiled_by" : "root",
  "compiled_in" : "7496ef0e0c91",
  "release_date" : "Fri Sep 14 12:24:06 UTC 2018",
  "doc" : "https://fiware-orion.readthedocs.org/en/master/"
}
}

same as @versatildefuy and @dherykw

[fgalan]

Please @mainakae could you also provide feedback on PR #3320 in order to see if the documentation modificaiton in that pull request is correct according your tests? Thanks!

[fgalan]

PRs https://github.com/telefonicaid/fiware-orion/pull/3379 and https://github.com/telefonicaid/fiware-orion/pull/3380 supposely fix this issue in a leak-safe way.

@jason-fox @versatildefuy @mainakae @dherykw could you please redeploy (using fiware:orion:latest) and confirm if it is working now, please? Thanks!

[jason-fox]

@fgalan - fiware\orion:latest is no longer showing the bug. It looks fixed to me.

[fgalan]

Thanks @jason-fox for the feedback. It would be great to have also the feedback from @versatildefuy @mainakae @dherykw but I understand your positive report is enough to close the issue.

As follow up actions I will:

• delete the fiware/orion:fix3162 image
• delete the branch bugfixing/3162_fix_but_potential_leak
• cancel PR #3320 as the documentation adjust is no longer meaningfull