Response to a GET request with attrs parameter contains unexpected properties.

[sekiguchi-yohei]

Hi.

I encountered an unexpected event in my verification using context registration.

The following is a description of what we tested.

①：POST /ngsi-ld/v1/csourceRegistrations/ 　Create the following context registrations. 　Succeeds in creating it.

{
  "@context": "https://fiware.github.io/tutorials.Step-by-Step/tutorials-context.jsonld",
  "id": "urn:ngsi-ld:ContextSourceRegistration:60d3e04c6abeb9e44004755b",
  "type": "ContextSourceRegistration",
  "endpoint": "http://orion-provider:1026/",
  "information": [
    {
      "entities": [
        {
          "id": "urn:ngsi-ld:Building:store001",
          "type": "Building"
        }
      ],
      "properties": [
        "weather"
      ]
    }
  ]
}

②：GET /ngsi-ld/v1/entities?id=urn:ngsi-ld:Building:store001&type=Building&attrs=location 　I expected only the id, type, and lolocation properties to be retrieved, but the response contains an unexpected property (weather).

[
  {
    "@context": "https://fiware.github.io/tutorials.Step-by-Step/tutorials-context.jsonld",
    "id": "urn:ngsi-ld:Building:store001",
    "type": "Building",
    "location": {
      "type": "GeoProperty",
      "value": {
        "type": "Point",
        "coordinates": [
          13.4447,
          52.5031
        ]
      }
    },
    ↓ here
    "weather": {
      "type": "Property",
      "value": "rain"
    }
  }
]

Why is this?

[sekiguchi-yohei]

Sorry, I forgot to mention the version. It occurs in the following version:

{
  "orionld version": "0.7.0",
  "orion version":   "1.15.0-next",
  "uptime":          "5 d, 19 h, 42 m, 19 s",
  "git_hash":        "nogitversion",
  "compile_time":    "Fri Mar 19 12:26:12 UTC 2021",
  "compiled_by":     "root",
  "compiled_in":     "buildkitsandbox",
  "release_date":    "Fri Mar 19 12:26:12 UTC 2021",
  "doc":             "https://fiware-orion.readthedocs.org/en/master/"
}

[kzangeli]

Seems to me you have found a bug in the experimental implementation of forwarding of Orion-LD. You are right, "weather" is not supposed to be present. Something that is "supposedly implemented and working" - you might have found a bug in it.

Forwarding is not supported yet in the NGSI-LD API. We're working on it but it will still take some time for it to be completely figured out (it's complex!) and added to the spec. Hopefully we'll have it defined and released in the NGSI-LD API spec by the end of 2021.

Jason Fox (iot-agents) and myself performed a PoC on forwarding some time ago, and that's possibly what you are seeing right now. It's unofficial, undocumented and simply just a proof of concept. However, we got it working for two operations (entity retrieval and patch of attributes) and your test should work as well.

Now, in order to understand what's really happening, I'd need more information.

1. Did you start your broker with the flag `-forwarding' set ?
2. Does the entity urn:ngsi-ld:Building:store001 live both in your local broker and the registered broker?
3. Please add the complete (well, only location and weather attrs are relevant) entities in the two brokers.

Like that I can set up my own test and understand what is happening. With some luck fix a bug! :)

Thank you for reporting!

[sekiguchi-yohei]

Thank you for your quick response!

1. Did you start your broker with the flag `-forwarding' set ?

Yes, I added '-forwarding' when starting the broker.

2. Does the entity urn:ngsi-ld:Building:store001 live both in your local broker and the registered broker?

Yes, urn:ngsi-ld:Building:store001 is present in both.

3. Please add the complete (well, only location and weather attrs are relevant) entities in the two brokers.

OK, the broker and provider entities will be ↓ respectively.

・broker

{
  "@context": "https://fiware.github.io/tutorials.Step-by-Step/tutorials-context.jsonld",
  "id": "urn:ngsi-ld:Building:store001",
  "type": "Building",
  "furniture": [
    {
      "type": "Relationship",
      "datasetId": "urn:ngsi-ld:Relationship:1",
      "object": "urn:ngsi-ld:Shelf:unit001"
    },
    {
      "type": "Relationship",
      "datasetId": "urn:ngsi-ld:Relationship:2",
      "object": "urn:ngsi-ld:Shelf:unit002"
    },
    {
      "type": "Relationship",
      "datasetId": "urn:ngsi-ld:Relationship:3",
      "object": "urn:ngsi-ld:Shelf:unit003"
    }
  ],
  "category": {
    "type": "Property",
    "value": "commercial"
  },
  "address": {
    "type": "Property",
    "value": {
      "streetAddress": "Bornholmer Straße 65",
      "addressRegion": "Berlin",
      "addressLocality": "Prenzlauer Berg",
      "postalCode": "10439"
    },
    "verified": {
      "type": "Property",
      "value": true
    }
  },
  "location": {
    "type": "GeoProperty",
    "value": {
      "type": "Point",
      "coordinates": [
        13.3986,
        52.5547
      ]
    }
  },
  "name": {
    "type": "Property",
    "value": "Bösebrücke Einkauf"
  }
}

・provider

{
  "@context": "https://fiware.github.io/tutorials.Step-by-Step/tutorials-context.jsonld",
  "id": "urn:ngsi-ld:Building:store001",
  "type": "Building",
  "weather": {
    "type": "Property",
    "value": "rain"
  }
}

[kzangeli]

Thank you for the info! Got me enough to be able to reproduce the error.

The bug is discovered and fixed, PR on its way (need to test everything first to make sure I haven't broken anything else ...)

This is the bug: The merge between registered attributes (weather) and the wanted attributes (location) is nothing. I simply didn't consider this possibility when I implemented the PoC. What the broker is doing currently when this happens (the bug), is that the forwarded request is sent without the attrs URL param - obtaining everything from the registered part. The bug fix is that when both registered attrs is there, and the URL param 'attrs' also, but the merge of the two gets nothing, then no forwarded request is to be sent. Easy fix, as almost always, once I have a way of reproducing the error.

Unless I run into unexpected problems, the PR should be merged before EOB tomorrow (it's getting late ... ;-))

Again, are you sure you want to play around with forwarding? :)

[kzangeli]

This is the PR: https://github.com/FIWARE/context.Orion-LD/pull/901. However, there are some problems with the docker base image and all tests fail.

We're moving from "debian stretch" to "ubuntu 20.04" as docker repo and quite a few changes have been necessary. This is why the tests are failing. I think this PR will have to wait until we finish the "ubuntu 20.04" task, which I hope and believe will be tomorrow.

[sekiguchi-yohei]

Thank you so much, Ken! Copy that. Testing awaits your success.

[kzangeli]

"which I hope and believe will be tomorrow" ...

Ubuntu 20.04 as base failed and we're now working with RedHat UBI (and got the same issues as with Ubuntu 20.04). We're still working on it, been at it for three weeks. Seeing some performance problems that we still cannot understand.

Just to let you know that you are not forgotten, we just ran into a massive amount of unexpected problems :(

[kzangeli]

With vacations coming, we postponed the base image change. The fix is merged and in develop, just ... still with debian stretch as distro.

Please test and let us know, by comments and/or closing of the issue.

[ Very sorry I had you waiting for so long ... it was "special circumstances" ... ]