Issue filtering entities with lazy attributes

[arturohu]

We're trying to run some filters on a type of entity. These entities are Device type and have static, lazy, and active attributes. We get the following response when we try to ask for the entities by satisfying a certain condition, for instance, "q=name=='Test Name'", and taking into account that "name" is a static attribute.

1. If the condition is satisfied for an entity
   • The entity will be displayed in the response as expected.
2. If the condition is not satisfied for an entity
   • The entity will be displayed in the response, but only showing the lazy attributes.
3. If the condition is not satisfied for any entity:
   • All those entities with only their lazy attributes are included in the response.

Examples:

Request (no entity satisfies the condition):

curl --location --request GET 'http://example/orion/v2/entities?options=dateModified&type=Device&q=fail==%27fail%27&attrs=lazyAttribute,activeAttribute' \
--header 'fiware-service: test' \
--header 'fiware-servicepath: /test'

Response:

[
   {
       "id": "urn:ngsi:Device:Test-Device1",
       "type": "Device",
       "lazyAttribute": {
           "type": "Number",
           "value": 10,
           "metadata": {}
       }
   },
   {
       "id": "urn:ngsi:Device:Test-Device2",
       "type": "Device",
       "lazyAttribute": {
           "type": "Number",
           "value": 20,
           "metadata": {}
       }
   },
   {
       "id": "urn:ngsi:Device:Test-Device3",
       "type": "Device",
       "lazyAttribute": {
           "type": "Number",
           "value": 30,
           "metadata": {}
       }
   },
]

Request (some entity satisfies the condition):

curl --location --request GET 'http://example/orion/v2/entities?options=dateModified&type=Device&q=name==%27Test%20Good%27&attrs=lazyAttribute,activeAttribute' \
--header 'fiware-service: test' \
--header 'fiware-servicepath: /test'

Response:

[
   {
       "id": "urn:ngsi:Device:Test-Device1",
       "type": "Device",
       "activeAttribute": {
           "type": "Number",
           "value": 0,
           "metadata": {}
       },
       "lazyAttribute": {
           "type": "Number",
           "value": 10,
           "metadata": {}
       }
       "dateModified": {
           "type": "DateTime",
           "value": "2020-07-21T12:23:34.00Z",
           "metadata": {}
       }
   },
   {
       "id": "urn:ngsi:Device:Test-Device2",
       "type": "Device",
       "lazyAttribute": {
           "type": "Number",
           "value": 20,
           "metadata": {}
       }
   },
   {
       "id": "urn:ngsi:Device:Test-Device3",
       "type": "Device",
       "lazyAttribute": {
           "type": "Number",
           "value": 30,
           "metadata": {}
       }
   },
]

I have tried it on Docker:

• mongo:3.6
• fiware/orion:2.2.0 and fiware/orion:2.4.0

[fgalan]

I understand (by the mention to "lazy attributes") that you are using IOTA in your setup. Is that correct? Which one and which version, please?

If you remove the registrations created by the IOTA (using DELETE /v2/registrations/<regId> operation), requests to Orion work as expected?

Thanks for the feedback!

[arturohu]

We're actually using an own context provider. When we ask Orion for the entities with attribute filter, this context provider receives the request and returns all the associated context elements. It seems that Orion doesn't apply the filter to all the entities returned by the context provider.

• Should the context provider handle the filter?
• Does Orion expect to receive all lazy attributes even if the filter isn't satisfied for the entity?

I have checked what happens when I remove the registrations associated with the devices. At the moment it has been deleted, Orion has responded as expected to the request of the entities with filter.

Thank you for helping us!

[floufen]

it looks like your issue is the same as #3657

[fgalan]

It is known issue that filtering is not working well with CPrs (see issue https://github.com/telefonicaid/fiware-orion/issues/2282) and maybe it is impating/causing the same issue you have here.

[arturohu]

Exactly, we are using CPrs registered using NGSIv1, and requesting entities with filtering through NGSIv2.

What will be the next step? Should we close this issue to attend the issue #2282 you have mentioned?

Thank you

[fgalan]

If you think #2282 covers this issue (and would do so with some comments to #2282), I think is a good idea to close this one.

After that, the next step would be to find some brave developer :) wanting to have a look at #2282 and solve the issue. I can guide on the process, if somebody commits with the task.

[mkavipriya]

Do we have a fix available for this filtering issue please? We are facing the same issue where Context providing returning invalid records and Orion forwarding them to the client as it is.

[fgalan]

Do we have a fix available for this filtering issue please? We are facing the same issue where Context providing returning invalid records and Orion forwarding them to the client as it is.

Already answered in issue https://github.com/telefonicaid/fiware-orion/issues/2282#issuecomment-746589644 ;)

What I said some time ago stills:

... the next step would be to find some brave developer :) wanting to have a look at #2282 and solve the issue. I can guide on the process, if somebody commits with the task.