telefonicaid / fiware-orion

Context Broker and CEF building block for context data management, providing NGSI interfaces.
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md
GNU Affero General Public License v3.0
210 stars 265 forks source link

Get /v2/entities?attrs=id don't work #3777

Closed fisuda closed 3 years ago

fisuda commented 3 years ago

'/v2/entities?attrs=id' doesn't seem to work in the latest version. If my understanding is mistaken, please tell me.

$ curl http://localhost:1026/version
{
"orion" : {
  "version" : "2.5.0-next",
  "uptime" : "0 d, 0 h, 0 m, 8 s",
  "git_hash" : "1744eeb6a3489c2c0bfbeceadaf471d0a3784a95",
  "compile_time" : "Fri Feb 12 14:55:26 UTC 2021",
  "compiled_by" : "root",
  "compiled_in" : "eea31cce414e",
  "release_date" : "Fri Feb 12 14:55:26 UTC 2021",
  "doc" : "https://fiware-orion.rtfd.io/",
  "libversions": {
     "boost": "1_53",
     "libcurl": "libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0",
     "libmicrohttpd": "0.9.70",
     "openssl": "1.0.2k",
     "rapidjson": "1.1.0",
     "mongodriver": "legacy-1.1.2"
  }
}
}

$ curl http://localhost:1026/v2/entities -H "Content-Type: application/json" --data '{"id":"id001"}'

$ curl http://localhost:1026/v2/entities
[{"id":"id001","type":"Thing"}]

$ curl http://localhost:1026/v2/entities?attrs=id
[]

$ curl --dump-header - "http://localhost:1026/v2/entities?attrs=id&options=count"
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 2
Content-Type: application/json
Fiware-Total-Count: 1
Fiware-Correlator: f4e21862-70fa-11eb-bf5e-0242c0a8800e
Date: Wed, 17 Feb 2021 08:34:31 GMT

[]
$ curl http://localhost:1026/version
{
"orion" : {
  "version" : "2.5.2",
  "uptime" : "0 d, 0 h, 1 m, 18 s",
  "git_hash" : "11e4cbfef30d28347162e5c4ef4de3a5d2797c69",
  "compile_time" : "Thu Dec 17 08:43:46 UTC 2020",
  "compiled_by" : "root",
  "compiled_in" : "5a4a8800b1fa",
  "release_date" : "Thu Dec 17 08:43:46 UTC 2020",
  "doc" : "https://fiware-orion.rtfd.io/en/2.5.2/",
  "libversions": {
     "boost": "1_53",
     "libcurl": "libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0",
     "libmicrohttpd": "0.9.70",
     "openssl": "1.0.2k",
     "rapidjson": "1.1.0",
     "mongodriver": "legacy-1.1.2"
  }
}
}

$ curl http://localhost:1026/v2/entities -H "Content-Type: application/json" --data '{"id":"id001"}'

$ curl http://localhost:1026/v2/entities
[{"id":"id001","type":"Thing"}]

$ curl http://localhost:1026/v2/entities?attrs=id
[{"id":"id001","type":"Thing"}]

Thanks

fgalan commented 3 years ago

I have one doubt, please:

Which is the intended use case for using GET /v2/entities?attrs=id? To get a list of just entity id and type for all entities, filtering out any attribute?

Thanks for the feedback!

fisuda commented 3 years ago

Thank you for your reply. I'm developing a command-line interface for NGSIv2.

The following command prints a list of IDs of all entities. It's like ls command. It uses GET /v2/entities?attrs=id.

$ ngsi list --host orion entities
airqualityobserved_0
airqualityobserved_1
airqualityobserved_2
airqualityobserved_3
airqualityobserved_4
airqualityobserved_5
airqualityobserved_6

This command prints id and attributes when running with --verbose,

$ ngsi list --host orion entities --verbose 
[{"id":"airqualityobserved_0","type":"AirQualityObserved","CO":{"type":"Number","value":325.891791155,"metadata":{}},"CO_Level":{"type":"Text","value":"low","metadata":{}},"NO":{"type":"Number","value":67.860296619,"metadata":{}},"NO2":{"type":"Number","value":51.948542112,"metadata":{}},"NOx":{"type":"Number","value":

The following command deletes all entities of certain entity type.

$ ngsi rm --host orion --type Thing

To get list of entities that will be deleted, I use GET /v2/entities?attrs=id&type={type} Even in this case, I can't get a list of IDs in the latest version.

$ curl http://localhost:1026/v2/entities -H "Content-Type: application/json" --data '{"id":"id001"}'

$ curl "http://localhost:1026/v2/entities?attrs=id&type=Thing"
[]

I want to reduce a size of the payload without getting unnecessary attributes.

Thank you.

fgalan commented 3 years ago

I think the change of behavior was introduced by PR https://github.com/telefonicaid/fiware-orion/pull/3751. In particular by this modification:

imagen

That modification was aimed at improving responses in some query forwarding scenarios but it seems (your feedback proves it) that also have a lateral effect impacting on how attrs= behave in scenarios without forwarding.

In fact, we should have detected it with the test/functionalTest/cases/2596_retrieve_entity_without_attrs/retrieve_entity_without_attrs.test and test/functionalTest/cases/2604_op_query_attrs/op_query_attrs.test tests, but we didn't realize (the PR includes modification in these tests and it shouldn't). Probably due to a missconcention between q=A (the parameters used to filter, i.e. to decide which entities to include in the response) and attrs=A (the parameter use to project, i.e. to decide which attributes, of the selected entities, are included in the response).

I'll have a look and try to modify the implementation, to keep the previous behavior with regards to attrs= in scenarios without forwarding and, at the same time, improve the responses to query forwarding in the intended way.

As side-note: it's interesting the usage you do of attrs=, using attrs=id. The NGSIv2 spec specifies that attrs= has to be a "comma-separated list of attribute names whose data are to be included in the response". But id is not a valid name for an attribute (from "JSON Entity Representation" section: "id and type are not allowed to be used as attribute names"). So with attrs=id you ensure that no attribute will match and you will always retrieve only the id and type of the entity. I think is a good solution by the time being, but maybe in the future we should define an special keyword to specify "I don't want any attribute" in a more explicit way, i.e. attrs=! (similar to the attrs=* we have now specify "all attributes").

kzangeli commented 3 years ago

attrs=! sounds like a good solution. In my opinion, if id (or any other 'non-attribute', like type or servicePath) is included in the attrs list of attribute names, the broker should give an error ...

fisuda commented 3 years ago

Thank you for the detailed explanation. I didn't understand that id is not a valid name for an attribute. My doubt was clear up.

fgalan commented 3 years ago

Thank you for the detailed explanation. I didn't understand that id is not a valid name for an attribute. My doubt was clear up.

While something like attrs=! gets implemented (I have create an issue about it: https://github.com/telefonicaid/fiware-orion/issues/3780), maybe a better alternative is to use something like attrs=__NONE which is a valid attr name (from the point of view of NGSIv2 spec rules) and will suffice for your use case on getting only entity ids and types in the response.

fgalan commented 3 years ago

Anyway, this fix in Orion is required:

I'll have a look and try to modify the implementation, to keep the previous behavior with regards to attrs= in scenarios without forwarding and, at the same time, improve the responses to query forwarding in the intended way.

Otherwise, the result with attrs=__NONE will be the same than with attrs=id (i.e. no entities in the respone).

fisuda commented 3 years ago

I updated my tool (NGSI Go). It uses GET /v2/entities?attrs=__NONE to get a list of entity IDs. Related PR: https://github.com/lets-fiware/ngsi-go/pull/88

fisuda commented 3 years ago

Cc: @jason-fox I sent a PR (https://github.com/FIWARE/tutorials.CRUD-Operations/pull/15) to update FIWARE/tutorials.CRUD-Operations.

fgalan commented 3 years ago

Fixed by PR https://github.com/telefonicaid/fiware-orion/pull/3783

fgalan commented 3 years ago

PR with the fix already merged in master.

@fisuda could you check with Orion latest (at commit 3984f9f or later) if the problem is now fixed and GET /v2/entities?attrs=__NONE correctly shows the id and type of all entities, please?

fisuda commented 3 years ago

GET /v2/entities?attrs=__NONE worked as expected. This issue has been resolved. I appreciate your prompt response to fix it.

$ ngsi --version
ngsi version 0.6.0-next (git_hash:30e8e5338f7f863420e4b06f0acee1b75e26efeb)

$ ngsi version --host orion
{
"orion" : {
  "version" : "2.5.0-next",
  "uptime" : "0 d, 0 h, 3 m, 39 s",
  "git_hash" : "3984f9fc30e90fa04682131ca4516b4d277eb27e",
  "compile_time" : "Mon Feb 22 17:37:11 UTC 2021",
  "compiled_by" : "root",
  "compiled_in" : "5e9c61ba627a",
  "release_date" : "Mon Feb 22 17:37:11 UTC 2021",
  "doc" : "https://fiware-orion.rtfd.io/",
  "libversions": {
     "boost": "1_53",
     "libcurl": "libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0",
     "libmicrohttpd": "0.9.70",
     "openssl": "1.0.2k",
     "rapidjson": "1.1.0",
     "mongodriver": "legacy-1.1.2"
  }
}
}

$ ngsi create entity --data '{"id":"id001","A":{"type":"Text","value":"FIWARE"}}'

$ ngsi list entities
id001

$ curl http://orion:1026/v2/entities?attrs=__NONE
[{"id":"id001","type":"Thing"}]

$ ngsi create entity --data '{"id":"id002","type":"Event","A":{"type":"Text","value":"FIWARE"}}'

$ ngsi list entities
id001
id002

$ ngsi list entities --verbose
[{"id":"id001","type":"Thing","A":{"type":"Text","value":"FIWARE","metadata":{}}},{"id":"id002","type":"Event","A":{"type":"Text","value":"FIWARE","metadata":{}}}]

$ ngsi rm --type Thing --run
1

$ ngsi list entities
id002

$ ngsi rm --type Event --run
1

$ ngsi list entities