search

telefonicaid / fiware-orion

Context Broker and CEF building block for context data management, providing NGSI interfaces.
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md
GNU Affero General Public License v3.0
210 stars 265 forks source link

NGSIv2 update forwarding sets attribute value to empty string #3647

Open michaeI-s opened 4 years ago

michaeI-s commented 4 years ago

Hi,

I'm trying to update the attribute of an entity that resides in a context provider via NGSIv2 update forwarding [Update request --> Orion v2 --> Orion v2 (CPr)]. Unfortunately the attribute is not updated with the new value from the request, but its value is set to an empty string.

This is my entity in the CPr:

{
    "id": "GtfsStop:test:MS",
    "type": "GtfsStop",
    "name": {
        "type": "Text",
        "value": "Rangsdorf, Bahnhof",
        "metadata": {
            "timestamp": {
                "type": "DateTime",
                "value": "2020-03-09T20:21:26.00Z"
            }
        }
    }
}

This is my PATCH request which is sent to the first Orion and subsequently forwarded to the CPr:

curl --location --request PATCH 'http://<ORION_HOST>/v2/entities/GtfsStop:test:MS/attrs' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": {
        "type": "Text",
        "value": "Rammsdorf, Bahnhof",
        "metadata": {
            "timestamp": {
                "type": "DateTime",
                "value": "2020-04-01T14:21:26.00Z"
            }
        }
    }
}'

And last but not least the CPr registration in the queried Orion:

{
    "id": "5e85b997493bcb6eac7e5c25",
    "description": "GtfsStop name provider for matching GtfsStop",
    "dataProvided": {
        "entities": [
            {
                "id": "GtfsStop:test:MS",
                "type": "GtfsStop"
            }
        ],
        "attrs": [
            "name"
        ]
    },
    "provider": {
        "http": {
            "url": "http://<CPr_HOST>/v2"
        },
        "supportedForwardingMode": "all",
        "legacyForwarding": true
    },
    "status": "active"
}

So my expectation is that the value of name is changed from "Rangsdorf, Bahnhof" to "Rammsdorf, Bahnhof" by this update. But it is just set to an empty string. However the timestamp in the corresponding metadata object is set to the new value.

Entity after update in the CPr:

{
    "id": "GtfsStop:test:MS",
    "type": "GtfsStop",
    "name": {
        "type": "Text",
        "value": "",
        "metadata": {
            "timestamp": {
                "type": "DateTime",
                "value": "2020-04-01T14:21:26.00Z"
            }
        }
    }
}

I also experimented with legacyForwarding property of the registration and tried to set its value to false, but it seems to be ignored and value flips back to true during creation (another bug?).

I'm testing with Orion v2.4 (both Query CB and CPr).

Kind regards, Michael

fgalan commented 4 years ago

According to CPrs and request forwarding documentation

On forwarding, any type of entity in the NGSIv2 update/query matches registrations without entity type. However, the opposite doesn't work, so if you have registrations with types, then you must use ?type in NGSIv2 update/query in order to obtain a match. Otherwise you may encounter problems, like the one described in this post at StackOverflow.

Your registration has types:

    "dataProvided": {
        "entities": [
            {
                "id": "GtfsStop:test:MS",
                "type": "GtfsStop"
            }
        ],

So maybe adding type to your update, that is change

curl --location --request PATCH 'http://<ORION_HOST>/v2/entities/GtfsStop:test:MS/attrs' \

to

curl --location --request PATCH 'http://<ORION_HOST>/v2/entities/GtfsStop:test:MS/attrs?type=GtfsStop' \

could help.

Could you try and tell us how it goes, please?

michaeI-s commented 4 years ago

Hi Fermín,

thanks for the hint. This is actually working - great!

I understand that ?type=<TYPE> is a secure way of addressing the right CPr registration in the broker, because practically there could be several registrations with different types matching the same update query - especially in connection with the idPattern (.*) matching feature.

But in a real world scenario a client usually does not necessarily know about CPr registrations and the general setup of the server-side architecture. He just sends a request to the query endpoint (probably without ?type=<TYPE>) and awaits a result. You wouldn't even be able to intercept and rewrite the request, because the PATCH request in this case provides no information about the addressed entity type. However, of course you could query the type for a named entity in the PATCH request prior to creating a new 'proxy request' from it. But this would notably increase the response time which is less than ideal. Also the idea of using a batch update operation as a workaround, consisting of only one entity with all its identification properties (id, type) is no option, because POST /op/update with action type append is merely mapping to PATCH /entities/<ENTITY_ID>/attrs.

So, if you do not want to create a huge amount of CPr registrations, each for every single entity that is created, for me the only solution at this time seems to be some kind of NGSI proxy between the client and the queried context broker. That proxy would have its own API spec that requires certain URL parameters to be given (e.g. ?type=<TYPE>). But this again would break the main idea of NGSI - having an uniform API across systems.

Do you have any idea for a smart solution?

Kind regards, Michael

fgalan commented 4 years ago

CB allows registrations without type. The need of specifying the type is only on queries/updates to be forwarded. As far as I remember, this is more a "gap" than a feature designed to enforce a secure way of addressing the right CPr registration (although at end it helps to that purpose of course :)

I'm afraid I don't have any smart solution for this at the present point, sorry... apart from having a look to the CB source code, analyze why this is happening and propose a solution in the form of pull request. Unfortunately, I don't have time to work on that line... but I'll be more than happy to provide assistance on the process if somebody in the community volunters to do that :)

fgalan commented 4 years ago

I did a quick search on opened issues and the following ones seems to be somehow related:

(Non exahustive search... there could be more related one I haven't found)

michaeI-s commented 4 years ago

Yes, actually these issues relate to this problem.

piwo1984 commented 3 years ago

Any progress on this issue? I agree with @michaeI-s that a client mostly does not know anything about the registrations.

fgalan commented 3 years ago

Any progress on this issue?

No progress as far as I know.

I have just added the "help wanted" label to make more explicit what I say in my previous comment about contributions on this coming from the community are welcome :)

ArqamFarooqui110719 commented 1 year ago

Hi @fgalan sir, I have just gone through these issue i.e. issues #2247, #3271. I have observed that currently there is no dirty update happening, instead I'm getting below error in place of dirty update:

< HTTP/1.1 404 Not Found
{"error":"NotFound","description":"The requested entity has not been found. Check type and id"}

So these issue can be closed?

I have followed below steps: Step-1: Run CP (orion as a CP) at port 1027

contextBroker -fg -logLevel INFO -logForHumans -harakiri -port 1027 -db orion-cpr1

Step-2: Created an entity on Orion-CP

curl -v localhost:1027/v2/entities -s -S -H 'Content-Type: application/json' -d @- <<EOF
> {
>     "id": "GtfsStop:test:MS",
>     "type": "GtfsStop",
>     "name": {
>         "type": "Text",
>         "value": "Rangsdorf, Bahnhof",
>         "metadata": {
>             "timestamp": {
>                 "type": "DateTime",
>                 "value": "2023-03-01T23:16:26.00Z"
>             }
>         }
>     }
> }
> EOF
*   Trying 127.0.0.1:1027...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1027 (#0)
> POST /v2/entities HTTP/1.1
> Host: localhost:1027
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 287
>
* upload completely sent off: 287 out of 287 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Connection: Keep-Alive
< Content-Length: 0
< Location: /v2/entities/GtfsStop:test:MS?type=GtfsStop
< Fiware-Correlator: 401b6aca-b859-11ed-bd97-f5e1f5b80b30
< Date: Wed, 01 Mar 2023 17:48:18 GMT
<

Step-3: Created registration of above entity as below:

curl -v localhost:1026/v2/registrations -s -S -H 'Content-Type: application/json' -H 'Accept: application/json' -d @-  <<EOF
> {
>     "description": "GtfsStop name provider for matching GtfsStop",
>     "dataProvided": {
>         "entities": [
>             {
>                 "id": "GtfsStop:test:MS",
>                 "type": "GtfsStop"
>             }
>         ],
>         "attrs": [
>             "name"
>         ]
>     },
>     "provider": {
>         "http": {
>             "url": "http://localhost:1027/v2"
>         }
>     }
> }
> EOF
*   Trying 127.0.0.1:1026...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1026 (#0)
> POST /v2/registrations HTTP/1.1
> Host: localhost:1026
> User-Agent: curl/7.68.0
> Content-Type: application/json
> Accept: application/json
> Content-Length: 365
>
* upload completely sent off: 365 out of 365 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Connection: Keep-Alive
< Content-Length: 0
< Location: /v2/registrations/63ff921f79af4921060ee35a
< Fiware-Correlator: 95baa62a-b85a-11ed-bd97-f5e1f5b80b30
< Date: Wed, 01 Mar 2023 17:57:51 GMT

Step-4: Get above created entity via request forwarding:

curl -vX GET 'localhost:1026/v2/entities?idPattern=.*&type=GtfsStop'
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1:1026...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1026 (#0)
> GET /v2/entities?idPattern=.*&type=GtfsStop HTTP/1.1
> Host: localhost:1026
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 177
< Content-Type: application/json
< Fiware-Correlator: 1e2afb2c-b85b-11ed-bd97-f5e1f5b80b30
< Date: Wed, 01 Mar 2023 18:01:40 GMT
<
* Connection #0 to host localhost left intact
[{"id":"GtfsStop:test:MS","type":"GtfsStop","name":{"type":"Text","value":"Rangsdorf, Bahnhof","metadata":{"timestamp":{"type":"DateTime","value":"2023-03-01T23:16:26.000Z"}}}}]

Step-5: update the entity attribute via PATCH with ?type, that was created on orion-CP at step-2`, entity successfully updated

curl --location --request PATCH 'http://localhost:1026/v2/entities/GtfsStop:test:MS/attrs?type=GtfsStop' --header 'Content-Type: application/json' \
> --data-raw '{
>     "name": {
>         "type": "Text",
>         "value": "test, Bahnhof",
>         "metadata": {
>             "timestamp": {
>                 "type": "DateTime",
>                 "value": "2023-03-01T23:31:26.00Z"
>             }
>         }
>     }
> }'
curl -vX GET 'localhost:1026/v2/entities?idPattern=.*&type=GtfsStop'                                       Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1:1026...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1026 (#0)
> GET /v2/entities?idPattern=.*&type=GtfsStop HTTP/1.1
> Host: localhost:1026
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 172
< Content-Type: application/json
< Fiware-Correlator: 35e62fca-b85b-11ed-bd97-f5e1f5b80b30
< Date: Wed, 01 Mar 2023 18:02:20 GMT
<
* Connection #0 to host localhost left intact
[{"id":"GtfsStop:test:MS","type":"GtfsStop","name":{"type":"Text","value":"test, Bahnhof","metadata":{"timestamp":{"type":"DateTime","value":"2023-03-01T23:31:26.000Z"}}}}]

Step-6: update the entity attribute via PATCH without ?type, that was created on orion-CP at step-2, getting error

curl -v --location --request PATCH 'http://localhost:1026/v2/entities/GtfsStop:test:MS/attrs' --header 'Content-Type: application/json' --data-raw '{
    "name": {
        "type": "Text",
        "value": "check, Bahnhof",
        "metadata": {
            "timestamp": {
                "type": "DateTime",
                "value": "2023-03-01T23:41:26.00Z"
            }
        }
    }
}'
*   Trying 127.0.0.1:1026...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1026 (#0)
> PATCH /v2/entities/GtfsStop:test:MS/attrs HTTP/1.1
> Host: localhost:1026
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 242
>
* upload completely sent off: 242 out of 242 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Connection: Keep-Alive
< Content-Length: 95
< Content-Type: application/json
< Fiware-Correlator: f5779e18-b85c-11ed-bd97-f5e1f5b80b30
< Date: Wed, 01 Mar 2023 18:14:51 GMT
<
* Connection #0 to host localhost left intact
{"error":"NotFound","description":"The requested entity has not been found. Check type and id"}

Please provide your opinion.

mapedraza commented 1 year ago

The expected behaviour is the one described in the documentation:

On forwarding, any type of entity in the NGSIv2 update/query matches registrations without entity type. However, the opposite doesn't work, so if you have registrations with types, then you must use ?type in NGSIv2 update/query in order to obtain a match. Otherwise you may encounter problems, like the one described in this post at StackOverflow.

According to the tests performed by @ArqamFarooqui110719, the issue seems it is not valid anymore and it is working as expexted:

@michaeI-s could you confirm the behaviour is solved? In that case, please close the Issue.