Subscriptions GET and DELETE

[tanktoo]

Hi, besides other problems with the NGSI-LD subscriptions (see #60 ) it seems there are some additional problems. If I create a subscription { "id": "urn:ngsi-ld:Subscription:testsubscription", "type": "Subscription", "entities": [ { "type": "vehicle" } ], "notification": { "format": "keyValues", "endpoint": { "uri": "http://mycallback:8080/callback" } }, "@context": "https://fiware.github.io/NGSI-LD_TestSuite/ldContext/testFullContext.jsonld" } I get a 201 Created back with location /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription. (So if I don't create it twice I get the right response without a crash.) If I try to get all subscriptions I it seems the interface is not implemented yet: { "type": "http://uri.etsi.org/ngsi-ld/errors/BadRequestData", "title": "not implemented - GET /ngsi-ld/v1/subscriptions", "details": "no details" } If I now want to avoid adding the subscription again I tried to delete it with DELETE http://ourorion:1026/ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription This results in { "type": "http://uri.etsi.org/ngsi-ld/errors/BadRequestData", "title": "The requested subscription has not been found. Check id", "details": "urn:ngsi-ld:Subscription:testsubscription" } This is strange as this is the same ID as used in the first step where I received the 201 Created.

[kzangeli]

Lots of stuff ... First, yes, GET /ngsi-ld/v1/subscriptions is not implemented. Then, it is weird that the GET individual subscription does not work. Very strange. I have functests in place that "make sure" that this works.

I will test this though, hopefully I get the "not found" as well ..

[tanktoo]

Didn't search for a specific id yet but GET with /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription also results n subscription not found. Detailed answer as it is slightly different from the DELETE one: { "type": "http://uri.etsi.org/ngsi-ld/errors/ResourceNotFound", "title": "subscription not found", "details": "urn:ngsi-ld:Subscription:testsubscription" }

[kzangeli]

Not sure why you are having problems with this but ... I assure you it works just fine ...

[tanktoo]

I think it is not found if I add fiware-servicepath and fiware-service to the header.

[kzangeli]

Hmm, if you use a different tenant (service) ... then it will of course not be found. The service path will do the same. Paste the exact request that you are using when not finding the subscription. I need the exact request for creating it as well. Like that perhaps we can get to the bottom of the problem.

[tanktoo]

Without the mentioned headers:

1. create "testsubscription" => works (201)
2. get "testsubscription" => works (200, description in body)
3. delete "testsubscription" => works(?) (got a 204 no content)
4. delete "testsubsription" again => seems 3. worked as I get a 404
5. create "testsubscription" again => broker crashes (maybe the subscrption is not deleted correctly? would then refer to issue60)

[kzangeli]

In the form of curl requests from command line would be perfect

[kzangeli]

OK, I will add Step 5 (create "testsubscription" again) to my functest of PR #71.

[tanktoo]

Seems again only a problem with postman... have tried all requests with curl and it works...

[kzangeli]

ok ... I can't help with postman, I'm afraid. Let's close this issue (and any other that only is affected by postman) and open a new one that describes better the problem.

[tanktoo]

With headers problem still exist that is does not find anything. subscription_issue67.txt

And yes don't know what postman is doing different... will avoid to use it. Besides this I attached the file with curl command with and without headers.

[kzangeli]

Reopening, for the Tenant/Service-Path issue

[kzangeli]

So, some background:

1. Service Path is not supported by orionld
2. Tenants are, but has still not been implemented

The service path, whatever you use, is stored in the database as "/" (to not complicate interoperability with "non-ld orion"). The tenant (fiware-service) right now is simply ignored. Shouldn't take me very long to implement, will put it high on my ToDo list.

That said, your test worked just fine for me, including the tenant and service-path in the requests. Will copy-paste the entire procedure and you can do the exact same test

[kzangeli]

First I start orionld -fg in a separate xterm. Then:

kz@xps:context.Orion-LD> subscription='{
2>     "id": "urn:ngsi-ld:Subscription:testsubscription",
2>     "type": "Subscription",
2>     "entities": [
2>         {
2>             "type": "vehicle"
2>         }
2>     ],
2>     "notification": {
2>         "format": "keyValues",
2>         "endpoint": {
2>             "uri": "http://mycallback.de:8080/callback"
2>         }
2>     },
2>     "@context": "https://fiware.github.io/NGSI-LD_TestSuite/ldContext/testFullContext.jsonld"
2> }'
kz@xps:context.Orion-LD> curl http://localhost:1026/ngsi-ld/v1/subscriptions -H "Content-Type: application/ld+json" -H 'fiware-service: openiot' -H 'fiware-servicepath: /' -d "$subscription"
kz@xps:context.Orion-LD> curl http://localhost:1026/ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription?prettyPrint=yes -H 'fiware-service: openiot' -H 'fiware-servicepath: /'
{
  "id": "urn:ngsi-ld:Subscription:testsubscription",
  "type": "Subscription",
  "entities": [
    {
      "type": "vehicle"
    }
  ],
  "notification": {
    "format": "keyValues",
    "endpoint": {
      "uri": "http://mycallback.de:8080/callback",
      "accept": "application/json"
    }
  },
  "expires": "2038-01-19T03:14:07Z",
  "@context": "https://fiware.github.io/NGSI-LD_TestSuite/ldContext/testFullContext.jsonld"
}
kz@xps:context.Orion-LD> curl -X DELETE http://localhost:1026/ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription
kz@xps:context.Orion-LD> curl http://localhost:1026/ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription?prettyPrint=yes -H 'fiware-service: openiot' -H 'fiware-servicepath: /'
{
  "type": "http://uri.etsi.org/ngsi-ld/errors/ResourceNotFound",
  "title": "subscription not found",
  "details": "urn:ngsi-ld:Subscription:testsubscription"
}

[kzangeli]

Tenants fixed in PR #74. Not that I think that will fix your problem, but you might give it a try. The docker image is still not ready, but hopefully tomorrow.

[kzangeli]

Any news on this?

[tanktoo]

Just did a quick test with my description in txt file. Seems its still not working.

[kzangeli]

Did you perform the test I proposed in https://github.com/FIWARE/context.Orion-LD/issues/67#issuecomment-492155194 ?

[tanktoo]

I get the following output from second curl command: { "type": "http://uri.etsi.org/ngsi-ld/errors/ResourceNotFound", "title": "subscription not found", "details": "urn:ngsi-ld:Subscription:testsubscription" }

[kzangeli]

Did you update the broker? What version do you have? (curl localhost:1026/ngsi-ld/ex/v1/version)

[tanktoo]

curl http://orionld:1026/ngsi-ld/ex/v1/version

{
  "branch": "develop",
  "kbase version": "0.2",
  "kalloc version": "0.2",
  "kjson version": "0.2"
}

[kzangeli]

I just found an error in my example. The broker doesn't support tenants by default - you need to start it with the -multiservice option for tenants to work. When started without that option, the broker ignores the Fiware-Service header and creates everything in the orion database.

So, use -multiservice when starting the broker - I imagine you already do this as you are using tenants. However, this doesn't change the outcome of the simple test above.

So, let's look deeper.

1. Before starting orionld, remove all mongo databases.
   • First list them: % echo "show dbs" | mongo --quiet
   • Then remove them (never mind 'admin', 'local', and 'config'): % echo 'db.dropDatabase()' | mongo orion % echo 'db.dropDatabase()' | mongo orion
   • Make sure they are all gone (except those three): % echo "show dbs" | mongo --quiet
2. Now start orionld -fg -multiservice (let's enable tenants this time)
3. Run the POST /ngsi-ld/v1/subscriptions against the broker
4. List the databases in mongo: % echo "show dbs" | mongo --quiet Now you should see one database called "orion-openiot". This is your tenant 'openiot'.
5. Now we ask mongo for the first subscription inside the tenant 'openiot': % echo "db.csubs.findOne()" | mongo orion-openiot

The output I get in step 5 is this:

kz@xps:context.Orion-LD> echo "db.csubs.findOne()" | mongo orion-openiot
MongoDB shell version v3.6.12
connecting to: mongodb://127.0.0.1:27017/orion-openiot?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("d8453392-c3f7-49ec-b3c9-4cd69cf04f00") }
MongoDB server version: 3.6.12
{
    "_id" : "urn:ngsi-ld:Subscription:testsubscription",
    "expiration" : NumberLong(2147483647),
    "reference" : "http://mycallback.de:8080/callback",
    "custom" : false,
    "mimeType" : "application/json",
    "throttling" : NumberLong(0),
    "servicePath" : "/",
    "status" : "active",
    "entities" : [
        {
            "id" : "",
            "isPattern" : "",
            "type" : "http://example.org/ngsi-ld/default/vehicle",
            "isTypePattern" : false
        }
    ],
    "attrs" : [ ],
    "metadata" : [ ],
    "blacklist" : false,
    "ldContext" : "https://fiware.github.io/NGSI-LD_TestSuite/ldContext/testFullContext.jsonld",
    "conditions" : [ ],
    "expression" : {
        "q" : "",
        "mq" : "",
        "geometry" : "",
        "coords" : "",
        "georel" : ""
    },
    "format" : "keyValues"
}
bye
kz@xps:context.Orion-LD> 

If you instead start the broker without the -multiservice option, the subscription will be created in the database orion.

What output do you get from step 5?

[kzangeli]

So, you got 'develop' as branch name ... of course. I recently automated the branch name creation and as you've checked out develop you get that name that says absolutely nothing. I may have to rethink this automation process :(

Show me your githash instead:

kz@xps:context.Orion-LD> git log
commit a34918346cd2769af643ebeac5288176b3adfe4b  ...

[tanktoo]

If I add "-multiservice" to docker-compose script it is not starting. How to show the githash? I use the docker version. There is no git installed within the container.

[kzangeli]

First of all, -multiservice is a CLI option for the broker. If we don't have git installed inside the docker, not much we can do ... I imagine the docker creation takes the last from develop and builds the docker image from it. Should be OK. I really don't know how to help you. You face errors that don't make a lot of sense ... Will ask for help from our docker specialist.

[caa06d9c]

Hi! -multiservice enabled by default in the container, this option included in the entry point, as well as fg, you don't need it in the docker-compose file. You can log in to the container without starting Orion with the option --entrypoint, like docker run -it --rm --entrypoint bash fiware/orion-ld and check version. Orion version in the latest image is correct 1.15.0-next (git version: a608977f893f1118018755986b3a5e3c84cd1387), it's a latest git hash.

[tanktoo]

root@0085b382d47a:/opt# orionld --version
1.15.0-next (git version: a608977f893f1118018755986b3a5e3c84cd1387)

[kzangeli]

ok, so far so good ... Now, let's play around a little more inside the container. Send the same requests of creation/retrieval/deletion of the subsciption and see what happens.

As Dmitrii says, the -multiservice option is enabled by default so that should not be a problem.

Perhaps the problem is that when you try to GET and DELETE the subscription you don't specify the tenant (with HTTP header Fiware-Service: XXX) ?

[kzangeli]

I just read the entire issue again. Had forgotten about postman. Does it only fail with postman?

[tanktoo]

No, I used curl with the commands you posted in 67.

[kzangeli]

ok ... Let's do the mongo exercise: https://github.com/FIWARE/context.Orion-LD/issues/67#issuecomment-496418569

The only explanation I can see is a problem with the tenants. This test will show us whether this is the case.

[tanktoo]

Deleted all docker images and started with compose. root@57308cf3c705:/# echo "show dbs" | mongo --quiet admin 0.000GB config 0.000GB iotagent-json 0.000GB local 0.000GB orion 0.000GB

Then after deleting databases: root@57308cf3c705:/# echo "show dbs" | mongo --quiet admin 0.000GB config 0.000GB local 0.000GB docker stop orion starting again with docker compose

curl http://localhost:1026/ngsi-ld/v1/subscriptions -H "Content-Type: application/ld+json" -H 'fiware-service: openiot' -H 'fiware-servicepath: /' -d "$subscription"

root@57308cf3c705:/# echo "show dbs" | mongo --quiet
admin          0.000GB
config         0.000GB
local          0.000GB
orion          0.000GB
orion-openiot  0.000GB

root@57308cf3c705:/# echo "db.csubs.findOne()" | mongo orion-openiot MongoDB shell version v3.6.12 connecting to: mongodb://127.0.0.1:27017/orion-openiot?gssapiServiceName=mongodb Implicit session: session { "id" : UUID("a8f9bb7a-2756-4a8a-8421-602c01250d87") } MongoDB server version: 3.6.12 { "_id" : "urn:ngsi-ld:Subscription:testsubscription", "expiration" : NumberLong(2147483647), "reference" : "http://mycallback.de:8080/callback", "custom" : false, "mimeType" : "application/json", "throttling" : NumberLong(0), "servicePath" : "/", "status" : "active", "entities" : [ { "id" : "", "isPattern" : "", "type" : "http://example.org/ngsi-ld/default/vehicle", "isTypePattern" : false } ], "attrs" : [ ], "metadata" : [ ], "blacklist" : false, "ldContext" : "https://fiware.github.io/NGSI-LD_TestSuite/ldContext/testFullContext.jsonld", "conditions" : [ ], "expression" : { "q" : "", "mq" : "", "geometry" : "", "coords" : "", "georel" : "" }, "format" : "keyValues" } bye

So far so fine. Now doing: curl http://localhost:1026/ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription?prettyPrint=yes -H 'fiware-service: openiot' -H 'fiware-servicepath: /'

{
  "type": "http://uri.etsi.org/ngsi-ld/errors/ResourceNotFound",
  "title": "subscription not found",
  "details": "urn:ngsi-ld:Subscription:testsubscription"
}

[kzangeli]

So, the subscription is there, but for some reason the broker can't find it ...

Let's check the broker traces. Have the broker started with -t 0-255 -logLevel DEBUG and post the entire log file (/tmp/orionld.log). Perhaps there we can get some clues ...

Also, remove all usages of service-path. orionld doesn't support service paths. It shouldn't fail because of it (they are ignored, but stored in mongo for backwards compatibility with orion), but as strange things are happening, we must try everything.

[tanktoo]

This should be the log from my last post. issue67.log

[kzangeli]

So, an extract of the traces:

Servicing NGSI-LD request 001: POST /ngsi-ld/v1/subscriptions fiware-service: openiot Creating error response: payload missing ((null))

Servicing NGSI-LD request 002: POST /ngsi-ld/v1/subscriptions fiware-service: openiot Got payload '{ "id": "urn:ngsi-ld:Subscription:testsubscription", "type": "Subscription", "entities": [ { "type": "vehicle" } ], "notification": { "format": "keyValues", "endpoint": { "uri": "http://mycallback.de:8080/callback" } }, "@context": "https://fiware.github.io/NGSI-LD_TestSuite/ldContext/testFullContext.jsonld" }' database name composed: 'orion-openiot' insert() in 'orion-openiot.csubs' collection: '{ _id: "urn:ngsi-ld:Subscription:testsubscription", ... Database Operation Successful (insert: { _id: "urn:ngsi-ld:Subscription:testsubscription", expiration: ...

Servicing NGSI-LD request 003: GET /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:testsubscription fiware-service: openiot database name composed: 'orion' query() in 'orion.csubs' collection: '{ _id: "urn:ngsi-ld:Subscription:testsubscription" }' subscription not found: 'urn:ngsi-ld:Subscription:testsubscription'

[kzangeli]

As you can see: Request 1 doesn't have any payload ... not sure what happened here. Never mind Request 2 creates the subscription in orion-openiot, as the Fiware-Service is 'openiot' Request 3 has the Fiware-Service set to 'openiot', but the lookup is done in the database 'orion' not 'orion-openiot' ! That's why it isn't found.

Seems like a bug to me. Will look into it. With this new data it shouldn't be too hard to find the bug

[tanktoo]

Request 1 was my fault, variable for subscription was empty. So no bug for request 1, sorry that I didn't mention.

[kzangeli]

No probs

[kzangeli]

Found the bug. Stupid mistake. Need a few hours to test the changes thoroughly. Will be merged into 'develop' later today (tonight) or tomorrow if my wife makes me cook dinner ... :)

@tanktoo, thank you very much for finding this bug for me, and very sorry I took so long to understand the problem ...

[kzangeli]

So, with PR #94 (just merged into develop), this issue should be fixed. Please test and let me know.

[tanktoo]

Working! :)

[kzangeli]

Great :)