Not all subscriptions are displayed via API

DasNordlicht commented 2 years ago

Versions Info Orion-LD:

"responseUrl": "http://orion-ld.fiware.svc:1026/ngsi-ld/ex/v1/version", "payload": { "Orion-LD version": "post-v1.0.1", "based on orion": "1.15.0-next", "kbase version": "0.8", "kalloc version": "0.8", "khash version": "0.8", "kjson version": "0.8.2", "microhttpd version": "0.9.75-0", "rapidjson version": "1.0.2", "libcurl version": "7.61.1", "libuuid version": "UNKNOWN", "mongocpp version": "1.1.3", "mongoc version": "1.22.0", "bson version": "1.22.0", "mongodb server version": "5.0.8", "boost version": "1_66", "openssl version": "OpenSSL 1.1.1k FIPS 25 Mar 2021", "branch": "", "cached subscriptions": 6, "Next File Descriptor": 40 }

The following 3 subscriptions are created and also visible in MongoDB:

urn:ngsi-ld:Subscription:3caaf73c-00ff-11ed-85c4-4a0947347d5c urn:ngsi-ld:Subscription:5bec9f9c-00ff-11ed-85c4-4a0947347d5c urn:ngsi-ld:Subscription:171c5f72-0101-11ed-a58e-9e4c25d95158

Via the endpoint /ngsi-ld/v1/subscriptions/ sometimes one or sometimes two subscriptions are displayed.

But with /ngsi-ld/v1/subscriptions/?options=count 3 are counted.

none of the subscriptions are triggered on new data and therefore there is no notification.

This is how the 3 subscribtions are in MongoDB

],
"blacklist" : false,
"name" : "Notify NR WeatherObserved Data",
"ldContext" : "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld",
"createdAt" : 1657533175.2848032,
"modifiedAt" : 1657533175.2848032,
"conditions" : [
    "https://uri.etsi.org/ngsi-ld/default-context/dateObserved"
],
"expression" : {
    "q" : "",
    "mq" : "",
    "geometry" : "",
    "coords" : "",
    "georel" : "",
    "geoproperty" : ""
},
"format" : "normalized"

],
"blacklist" : false,
"name" : "Notify QL WeatherObserved Data",
"ldContext" : "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld",
"createdAt" : 1657533227.7244992,
"modifiedAt" : 1657533227.7244992,
"conditions" : [
    "https://uri.etsi.org/ngsi-ld/default-context/dateObserved"
],
"expression" : {
    "q" : "",
    "mq" : "",
    "geometry" : "",
    "coords" : "",
    "georel" : "",
    "geoproperty" : ""
},
"format" : "normalized"

],
"blacklist" : false,
"name" : "Notify QL WeatherObserved Data Linked",
"ldContext" : "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld",
"createdAt" : 1657533971.268081,
"modifiedAt" : 1657533971.268081,
"conditions" : [
    "https://uri.etsi.org/ngsi-ld/default-context/dateObserved"
],
"expression" : {
    "q" : "",
    "mq" : "",
    "geometry" : "",
    "coords" : "",
    "georel" : "",
    "geoproperty" : ""
},
"format" : "normalized"

} `

kzangeli commented 2 years ago

ok ... Could you pass the requests creating the subscriptions (instead of showing how they end up in mongo)? I'm especially interested in the Entity Id "WeatherObserved", You might have found a bug somewhere in the creation of subscriptions ...

I'd like to try to reproduce the thing and fix it ! I imagine you just do a "GET /ngsi-ld/v1/subscriptions" So, complete requests, with HTTP headers and all would really help.

Also (next step), info on the entity creations/updates that are supposed to trigger notifications would be good to have.

DasNordlicht commented 2 years ago

The Subscribtion POST and GET

`curl --location --request POST 'https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/' \ --header 'fiware-service: captn' \ --header 'fiware-servicepath: /' \ --header 'Content-Type: application/ld+json' \ --header 'Authorization: Bearer TOKEN \ --data-raw '{ "description": "QL WeatherObserved", "type": "Subscription", "name": "Notify QL WeatherObserved Data Linked", "entities": [ { "idPattern": "WeatherObserved", "type": "WeatherObserved" } ], "watchedAttributes": [ "dateObserved" ], "isActive": true, "notification": { "attributes": [ "location", "atmosphericPressure", "dateObserved", "feelLikesTemperature", "gustSpeed", "relativeHumidity", "temperature", "visibility", "weatherType", "windDirection", "windSpeed", "address" ], "format": "normalized", "endpoint": { "uri": "http://quantumleap-quantumleap.fiware.svc:8086/v2/notify", "accept": "application/json" } }, "@context":"https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld" }'

curl --location --request GET 'https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/?options=count' \ --header 'fiware-service: captn' \ --header 'fiware-servicepath: /' \ --header 'Authorization: Bearer TOKEN'`

The Entity POST

curl --location --request POST 'https://apis.captn.stag.addix.io/ngsi-ld/v1/entityOperations/upsert/?options=update' \ --header 'fiware-service: captn' \ --header 'fiware-servicepath: /' \ --header 'Content-Type: application/ld+json' \ --header 'Accept: application/ld+json' \ --header 'Authorization: Bearer TOKEN' \ --data-raw '[{ "id": "urn:ngsi-ld:WeatherObserved:ADDIX:owm:54381016", "type": "WeatherObserved", "dateObserved": { "type": "Property", "value": { "@type": "DateTime", "@value": "2022-07-11T17:12:22.000Z" } }, "atmosphericPressure": { "type": "Property", "value": 1022 }, "relativeHumidity": { "type": "Property", "value": 52 }, "feelLikesTemperature": { "type": "Property", "value": 19.83 }, "temperature": { "type": "Property", "value": 20.38 }, "visibility": { "type": "Property", "value": 10000 }, "windDirection": { "type": "Property", "value": 50 }, "windSpeed": { "type": "Property", "value": 2.06 }, "gustSpeed": { "type": "Property", "value": 0 }, "weatherType": { "type": "Property", "value": "Clear" }, "source": { "type": "Property", "value": "https://openweathermap.org" }, "@context": ["https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld"] }]'

kzangeli commented 2 years ago

Just a few comments:

fiware-service - that's NGSIv2 - use NGSILD-Tenant instead (fiware-service is still supported for backwards compatibility but ...)
fiware-servicepath - that's also NGSIv2 - not supported in NGSI-LD. We recently added "Scope" to the NGSI-LD API spec, which is a replacement for NGSIv2's Service Path. Very recently. This is not yet supported by Orion-LD.

So, I'd fix those two details. Not that I believe it will fix your problem, that's for me to look into. But, you should amend the headers anyway, and especially bear in mind that the NGSIv2 Service Path is not supported by any NGSI-LD broker.

Also, please don't attempt to send any core context to the broker. The core context is already built into the brokers and not necessary. That is part of the NGSI-LD API. If you have a user context you want to use, then please send it. Sending core contexts will just make the requests slower, as the broker needs to examine it, decide it's a core context and then discard it.

Now, you said you had three subscriptions, but in your previous comment you only list one of them. I'll need all three to debug this problem you're seeing (problem no 1)

DasNordlicht commented 2 years ago

The only difference between the other two is the Notification URL and watchedAttributes:

1. Subscribtion:

`curl --location --request POST 'https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/' --header 'fiware-service: captn' --header 'fiware-servicepath: /' --header 'Content-Type: application/ld+json' --header 'Authorization: Bearer TOKEN --data-raw '{ "description": "NR WeatherObserved", "type": "Subscription", "name": "Notify NR WeatherObserved Data", "entities": [ { "idPattern": "WeatherObserved", "type": "WeatherObserved" } ], "watchedAttributes": [ "dateObserved" ], "isActive": true, "notification": { "attributes": [ "location", "atmosphericPressure", "dateObserved", "feelLikesTemperature", "gustSpeed", "relativeHumidity", "temperature", "visibility", "weatherType", "windDirection", "windSpeed", "address" ], "format": "normalized", "endpoint": { "uri": "http://node-red-captn.captn.scv/fromcb", "accept": "application/json" } }, "@context":"https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld" }'

2. Subscribtion

`curl --location --request POST 'https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/' --header 'fiware-service: captn' --header 'fiware-servicepath: /' --header 'Content-Type: application/ld+json' --header 'Authorization: Bearer TOKEN --data-raw '{ "description": "QL WeatherObserved", "type": "Subscription", "name": "Notify QL WeatherObserved Data Linked", "entities": [ { "idPattern": "WeatherObserved", "type": "WeatherObserved" } ], "watchedAttributes": [ "https://uri.etsi.org/ngsi-ld/default-context/dateObserved" ], "isActive": true, "notification": { "attributes": [ "location", "atmosphericPressure", "dateObserved", "feelLikesTemperature", "gustSpeed", "relativeHumidity", "temperature", "visibility", "weatherType", "windDirection", "windSpeed", "address" ], "format": "normalized", "endpoint": { "uri": "http://quantumleap-quantumleap.fiware.svc:8086/v2/notify", "accept": "application/json" } }, "@context":"https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld" }'

The 3rd subscribtion is the one above

`curl --location --request POST 'https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/' --header 'fiware-service: captn' --header 'fiware-servicepath: /' --header 'Content-Type: application/ld+json' --header 'Authorization: Bearer TOKEN --data-raw '{ "description": "QL WeatherObserved", "type": "Subscription", "name": "Notify QL WeatherObserved Data Linked", "entities": [ { "idPattern": "WeatherObserved", "type": "WeatherObserved" } ], "watchedAttributes": [ "dateObserved" ], "isActive": true, "notification": { "attributes": [ "location", "atmosphericPressure", "dateObserved", "feelLikesTemperature", "gustSpeed", "relativeHumidity", "temperature", "visibility", "weatherType", "windDirection", "windSpeed", "address" ], "format": "normalized", "endpoint": { "uri": "http://quantumleap-quantumleap.fiware.svc:8086/v2/notify", "accept": "application/json" } }, "@context":"https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.5.jsonld" }'

However, I'll try again now with the improvements from your comments above. So with NGSILD-Tenant and without fiware-service as well as without a core context specification.

kzangeli commented 2 years ago

OK, now I believe I have all the info I need to look into this. About my recommendations ... yeah, you should definitely follow them, but, I don't think it will solve your immediate issues. Seems to be up to me to solve that :) On a trip this week (Brussels Airport right now), but, I find time, every now and then ... This one and two more issues are on the very top of my ToDo. So, don't despair ;)

kzangeli commented 2 years ago

So, first comment:

"idPattern": "WeatherObserved"

That's part of the subscriptions. I don't check the value of "idPattern", simply cause ... it's too darn difficult. BUT, did you mean ".*WeatherObserved" ? (anything ending in WeatherObserved) Cause, only "WeatherObserved", that will ONLY match the exact value "WeatherObserved" and as that's not a URI, no entity can ever have that ID => the subscription will never ever trigger notifications.

So, fix that. I'd just remove ' idPattern` if I were you - whenever with problems, simplify. Then build back up again once it's working.

I have a simple functest creating your first subscription, and I have no problem retrieving it with a GET /subscriptions - make sure you are using the tenant in the GET request. Wait, you said you sometimes got one sometimes 2 ... ok, scratch that - might be a bug related to the URL parameter "limit" - I'll check that.

kzangeli commented 2 years ago

As a footnote: In my test I changed the idPattern to "urn:WeatherObserved" and then I created an entity with that exact id and the notification arrived.

DasNordlicht commented 2 years ago

So, first comment:
"idPattern": "WeatherObserved"
That's part of the subscriptions. I don't check the value of "idPattern", simply cause ... it's too darn difficult. BUT, did you mean ".*WeatherObserved" ? (anything ending in WeatherObserved) Cause, only "WeatherObserved", that will ONLY match the exact value "WeatherObserved" and as that's not a URI, no entity can ever have that ID => the subscription will never ever trigger notifications.

Do I understand correctly that I have to specify the appropriate RegEx for idPattern?

kzangeli commented 2 years ago

If I were you, to start with, I'd just remove the idPattern - accept any entity id - filter only on entity type. Until things start to work. Then add idPattern back, but, you'll have to fix the REGEX - can't be just "WeatherObserved".

DasNordlicht commented 2 years ago

If I were you, to start with, I'd just remove the idPattern - accept any entity id - filter only on entity type. Until things start to work. Then add idPattern back, but, you'll have to fix the REGEX - can't be just "WeatherObserved".

ok, the subscriptions work, trigger and send their notifications. Only the visibility at GET https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/?options=count is still a problem.

kzangeli commented 2 years ago

ok, good. I'm trying to reproduce that error with GET /subs but still no success. Still in our All-Hands meeting, so I don't have too much time, I might come back to you asking for more info, in my attempts to reproduce.

kzangeli commented 2 years ago

I seem unable to reproduce, on my Ubuntu 20.04. Sometimes bugs appear on one distro but not on others. Can depend on a variety of reasons. One thing that would produce the strange thing you are experiencing would be a failure to initialize the pagination variables. I tried different combinations but it all worked just fine for me.

So, I'll ask you to run your "GET /subs" with the following URL-parameters:

GET /ngsi-ld/v1/subscriptions?offset=0&limit=3

You have 3 subscriptions, right? (not that it really matters) If the problem goes away for you, then I'll know exactly where to look.

DasNordlicht commented 2 years ago

ok, I think we have found the error. We are running the Orion-LD on a HA Kubernetes cluster and have 2 instances of the Orion-LD running and also the MongoDB is running with 3 instances. Apparently one instance had a problem which affected the request to MongoDB. After we restarted both Orion-LD PODs, all queries always returned the 3 existing subscriptions. We have now observed this for a longer time after the restart and the error has not occurred again. Unfortunately, we could not see any indication of a problem in the logs, the instances looked clean on their own. We continue to monitor the issue.

Thank you for your support so we have now been able to learn quite a bit about the Orion LD.

kzangeli commented 2 years ago

ok, I understand. It actually explains everything and I should have suspected this from the very start I feel I should explain to you in detail how this works. It's a quite special setup to run various brokers against one single database.

If you now create another subscription, your problem is back, until you restart your brokers.

If you want to, we can have a call and I'll explain everything in detail - ken.zangelin@fiware.org

kzangeli commented 2 years ago

This is what I want to explain, in detail: https://fiware-orion.readthedocs.io/en/master/admin/perf_tuning.html#subscription-cache

Fully working in Orion - 100% untested in Orion-LD (it just might work). Haven't gotten to that point yet. You are the first to run into troubles. But, your troubles are for not using the sub propagation mechanism (it's off by default in Orion-LD). That's why you see different sets of subscriptions in different instances of the broker - no propagation.

kzangeli commented 2 years ago

So, I tried the subscription propagation in Orion-LD. It was actually tested already, as I do have a number of functests where I create subscriptions, kill+restart the broker and make sure the subs from the DB are correctly inserted in the sub-cache on startup. It's the exact same function that is used for propagation.

This here is the test I ran:

PRE;
# 01. Start Context Broker 1 (CB1) with -subCacheIval 2
# 02. Start Context Broker 2 (CB2) with -subCacheIval 2 and against the same mongo DB as CB1

TEST: 
# 01. Create subscription S1 on CB1
# 02. Create subscription S2 on CB2
# 03. GET all subscriptions from CB1 - see S1
# 04. GET all subscriptions from CB2 - see S2
# 05. sleep 2 secs - to let the subscriptions propagate
# 06. GET all subscriptions from CB1 - see S1 and S2
# 07. GET all subscriptions from CB2 - see S2 and S1

You can see it in the PR #1184, the test is called ngsild_subscription-cache-propagation.test

What's missing for you is to start the brokers with the CLI -subCacheIval <no of seconds>.

FR-ADDIX commented 2 years ago

ok we will test it ...

kzangeli commented 2 years ago

Remember, the time of "subscription propagation" between brokers is crucial. If you try to GET subscriptions before the propagation has been done, you will not get the desired result.

Personally, this is not how I would implement the whole propagation thing. My old bosses at Telefonica R&D made me implement it this way. I protested but ... also wanted to keep my job ... :))

FR-ADDIX commented 2 years ago

ok we set -subCacheIval 60

we now get the following errors in the logs:

The subscription was created like this:

curl --location --request POST 'https://apis.captn.stag.addix.io/ngsi-ld/v1/subscriptions/' \
--header 'Fiware-Service: captn' \
--header 'NGSILD-Tenant: captn' \
--header 'Authorization: Bearer TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
    "description": "QL Vehicle",
    "type": "Subscription",
    "name": "Notify QL Vehicle Data",
    "entities": [{
        "type": "Vehicle"
    }],
    "watchedAttributes": [
        "dateObserved"
    ],
    "isActive": true,
    "notification": {
        "attributes": [
        ],
        "format": "normalized",
        "endpoint": {
            "uri": "http://quantumleap-quantumleap.fiware.svc:8668/v2/notify",
            "accept": "application/json",
            "receiverInfo": [{
                "key": "Fiware-Service",
                "value": "captn"
                }]
        }
    }
}'

kzangeli commented 2 years ago

OK, that error message should probablt not appear in the log -. seems like a "false positive". You have created a subscription in the tenant "captn", and then you try to retrieve it without specifying the tenant? You won't find it that way. It doesn't exist under the default tenant.

I just assume this is what is happening.

Try to tretrieve the subscription specifying the tenant 'captn'. It that doesn't work, then there is a bug

kzangeli commented 2 years ago

I looked a bit closer. The error comes from after an entity creation/update (and a notification), trying to modify the stat-counters/timestamps of the subscription.

I need no more input, it seems like a bug provoked by my latest "fixes". I'll get right on that, let you know once I have it fixed,

kzangeli commented 2 years ago

What was missing was to assign the tenant to the subscription in a special case. Thanks to you I got aware of the problem, fixed it, and added a functest for it, so that I will know if it ever reappears.

Thanks!

The issue should be fixed (at least my test now works :)), so, please try it and close the issue if all is OK. The dockerhub tag seems to be 1.1.0-PRE-980.

I saw you used "latest" - don't do that, it's quite risky. Instead, find a tag that works for you and don't change it until you need something that is not implemented/working. Only after something is fixed, change the tag.

"latest" changes normally a few times every day, and there is always a risk that something you need stops working after I "fix things" ... So, don't use "latest".

DasNordlicht commented 2 years ago

We have now tested the 1.1.0-PRE-980 and no longer see any error.

Thank you for the fast and friendly support.

By the way, we did not use the latest version but the version 1.1.0-PRE-979. The version info via API seems to provide a different version information ;-)

"Orion-LD version": "post-v1.0.1" is not the version number used We have used the 1.1.0-PRE-980 ;-)

{ "Orion-LD version": "post-v1.0.1", "based on orion": "1.15.0-next", "kbase version": "0.8", "kalloc version": "0.8", "khash version": "0.8", "kjson version": "0.8.2", "microhttpd version": "0.9.75-0", "rapidjson version": "1.0.2", "libcurl version": "7.61.1", "libuuid version": "UNKNOWN", "mongocpp version": "1.1.3", "mongoc version": "1.22.0", "bson version": "1.22.0", "mongodb server version": "5.0.8", "boost version": "1_66", "openssl version": "OpenSSL 1.1.1k FIPS 25 Mar 2021", "branch": "", "cached subscriptions": 3, "Next File Descriptor": 41 }

kzangeli commented 2 years ago

ok, good! Seems like I mixed issues in my head (got 3 external issues at the same time, while in Berlin). I based my assumption of using "latest" on a docker-compose file. Not yours :) My bad. Good that you're not using "lñatest" - lots and lotsd of people do and ... it's simply not a good idea.

Happy that everything worked out in the end. Sorry about the late reaction from my side, but I was out elsewhere and just didn't have the opportunity to get on it right away

FIWARE / context.Orion-LD

Not all subscriptions are displayed via API #1179