Problems with showChanges notification parameter for periodic subscription operations

[daniel-gonzalez-sanchez]

Hi everyone,

I was playing again with NGSI-LD subscriptions in order to check the resolution of previously closed issues #522 and #523 regarding sysAttrs and showChanges notification parameters in subscription operations when I discovered a new issue. I've discovered that the showChanges notification parameter works for on-change subscriptions but does not work for periodic subscriptions.

Let me show an example:

Environment

• scorpiobroker/all-in-one-runner:java-latest (using the sample docker-compose file found in the official Scorpio Context Broker GitHub repository)

Steps to reproduce Creating a subscription based on changes to a sample TemperatureSensor NGSI-LD Entity with the showChanges and sysAttrs notification parameters:

curl --location --request POST 'localhost:9090/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data-raw '{
    "id": "urn:ngsi-ld:Subscription:OnChange:TemperatureSensor",
    "type": "Subscription",
    "entities": [{
        "type": "TemperatureSensor"
     }],
    "description": "On-change subscription to TemperatureSensor entities for changes within temperature property.",
    "watchedAttributes": ["temperature"],
    "notification": {
           "endpoint": {
           "uri": "http://consumer:8082/notify",
           "accept": "application/json"
           },
           "attributes": ["temperature"],
           "showChanges": true,
           "sysAttrs": true
    }
}'

If we request all the committed subscriptions with the following query:

curl --location 'http://localhost:9090/ngsi-ld/v1/subscriptions/' \
--header 'Accept: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' 

, the regarding response body is the following:

[
    {
        "id": "urn:ngsi-ld:Subscription:OnChange:TemperatureSensor",
        "type": "Subscription",
        "description": "On-change subscription to TemperatureSensor entities for changes within temperature property.",
        "entities": [
            {
                "type": "TemperatureSensor"
            }
        ],
        "isActive": true,
        "notification": {
            "endpoint": {
                "accept": "application/json",
                "uri": "http://notifier-consumer:8082/notify"
            },
            "format": "normalized",
            "attributes": ["temperature"],
            "showChanges": true,
            "sysAttrs": true
        },
        "timesFailed": 0,
        "timesSent": 0,
        "watchedAttributes": [
            "temperature"
        ],
        "status": "active"
    }
]

Creating a sample TemperatureSensor NGSI-LD Entity with a temperature Property that includes an unitCode attribute as additional metadata to indicate the unit of measurement of the temperature value:

curl --location 'http://localhost:9090/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
    "id": "urn:ngsi-ld:TemperatureSensor:1",
    "type": "TemperatureSensor",
    "temperature": {
        "type": "Property",
        "value": 27.9,
        "unitCode": "CEL"
    }
}'

, the first notification received by the consumer is the following one:

{'id': 'urn:ngsi-ld:TemperatureSensor:1', 'type': 'TemperatureSensor', 'temperature': {'type': 'Property', 'createdAt': '2024-09-08T12:13:53.921000Z', 'previousValue': '@none', 'value': 27.9, 'modifiedAt': '2024-09-08T12:13:53.921000Z', 'observedAt': '2024-09-08T12:13:53.633Z', 'unitCode': 'CEL'}, 'createdAt': '2024-09-08T12:13:53.921000Z', 'modifiedAt': '2024-09-08T12:13:53.921000Z'}

The notification includes the createdAt and modifiedAt attributes thanks to the sysAttrs notification parameter, and the unitCode attribute is included within the temperature Property. And if we commit a Partial Attribute Update for the temperature Property with a new temperature value:

curl --location --request PATCH 'http://localhost:9090/ngsi-ld/v1/entities/urn:ngsi-ld:TemperatureSensor:1/attrs/temperature' \
--header 'Content-Type: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
    "type":"Property", 
    "value": 29.9, 
    "unitCode": "CEL"
}'

, the second notification received by the consumer is the following one:

{'id': 'urn:ngsi-ld:TemperatureSensor:1', 'type': 'TemperatureSensor', 'temperature': {'type': 'Property', 'previousValue': 27.9, 'value': 39.9, 'modifiedAt': '2024-09-08T12:14:03.765000Z', 'unitCode': 'CEL'}, 'createdAt': '2024-09-08T12:13:53.921000Z', 'modifiedAt': '2024-09-08T12:14:03.765000Z'}

As we can see, the behavior is as expected for the on-change subscription with the showChanges notification parameter and we get in the notification both the previous and current value of the temperature Property.

Let's repeat the process with a periodic subscription. Creating a periodic subscription to a sample TemperatureSensor NGSI-LD Entity with a timeInterval parameter equal to 10 seconds and with the showChanges and sysAttrs notification parameters configured:

curl --location --request POST 'localhost:9090/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data-raw '{
    "id": "urn:ngsi-ld:Subscription:TemperatureSensor",
    "type": "Subscription",
    "entities": [{
        "type": "TemperatureSensor"
     }],
    "description": "Periodic subscription to TemperatureSensor entities for their temperature property.",
    "timeInterval": 10,
    "notification": {
           "endpoint": {
           "uri": "http://consumer:8082/notify",
           "accept": "application/json"
           },
           "attributes": ["temperature"],
           "showChanges": true,
           "sysAttrs": true
    }
}'

If we request all the committed subscriptions with the following query:

curl --location 'http://localhost:9090/ngsi-ld/v1/subscriptions/' \
--header 'Accept: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' 

, the regarding response body is the following:

[
    {
        "id": "urn:ngsi-ld:Subscription:Periodic:TemperatureSensor",
        "type": "Subscription",
        "description": "Periodic subscription to TemperatureSensor and HumiditySensor entities.",
        "entities": [
            {
                "type": "TemperatureSensor"
            }
        ],
        "isActive": true,
        "notification": {
            "attributes": "temperature",
            "endpoint": {
                "accept": "application/json",
                "uri": "http://scorpio-notifier-consumer-periodic:8084/notify"
            },
            "format": "normalized",
            "showChanges": true,
            "sysAttrs": true
        },
        "timeInterval": 10,
        "timesFailed": 0,
        "timesSent": 0,
        "status": "active"
    }
]

Creating a sample TemperatureSensor NGSI-LD Entity with a temperature Property that includes an unitCode attribute as additional metadata to indicate the unit of measurement of the temperature value:

curl --location 'http://localhost:9090/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
    "id": "urn:ngsi-ld:TemperatureSensor:2",
    "type": "TemperatureSensor",
    "temperature": {
        "type": "Property",
        "value": 27.9,
        "unitCode": "CEL"
    }
}'

, the first notification received by the consumer is the following one:

{'id': 'urn:ngsi-ld:TemperatureSensor:2', 'type': 'TemperatureSensor', 'temperature': {'type': 'Property', 'createdAt': '2024-09-08T13:29:19.739000Z', 'value': 27.9, 'modifiedAt': '2024-09-08T13:29:19.739000Z', 'unitCode': 'CEL'}, 'createdAt': '2024-09-08T13:29:19.739000Z', 'modifiedAt': '2024-09-08T13:29:19.739000Z'}

The notification includes the createdAt and modifiedAt attributes thanks to the sysAttrs notification parameter, and the unitCode attribute is included within the temperature Property. But for this first notification, it does not include the previousValue: @none attribute. And if we commit a Partial Attribute Update for the temperature Property with a new temperature value before the 10-seconds period of the following notification is met:

curl --location --request PATCH 'http://localhost:9090/ngsi-ld/v1/entities/urn:ngsi-ld:TemperatureSensor:1/attrs/temperature' \
--header 'Content-Type: application/json' \
--header 'Link: <https://fiware.github.io/data-models/full-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '{
    "type":"Property", 
    "value": 29.9, 
    "unitCode": "CEL"
}'

, the second notification received by the consumer is the following one:

{'id': 'urn:ngsi-ld:TemperatureSensor:1', 'type': 'TemperatureSensor', 'temperature': {'type': 'Property', 'createdAt': '2024-09-08T13:29:19.739000Z', 'value': 29.9, 'modifiedAt': '2024-09-08T13:29:25Z', 'unitCode': 'CEL'}, 'createdAt': '2024-09-08T13:29:19.739000Z', 'modifiedAt': '2024-09-08T13:29:25Z'}

As you can see, the behavior is not as expected for the periodic subscription with the showChanges notification parameter since the notification does not appear with the previous value (i.e., the previousValue attribute) inside the temperature Property.

We can also observe how, for some reason, in the second notification the modifiedAt system-generated attribute no longer has more than seconds precision instead of milliseconds. Also, in this case, unlike with on-change subscriptions, the createdAt system-generated attribute appears both inside and outside the temperature Property in the notifications (this issue is also discussed in open issue #608).

Thanks again!

Dani

[martin-p-bauer]

Hi Dani,

I just checked the specification and it is not as explicit as I thought regarding the explanation of the showChanges feature. The original intention was to show what change triggered the notification, i.e. only the triggering attribute would have a previousX subproperty. With this assumption, it is clear that you do not see a previousValue in a notification to a periodic subscription as notifications are triggered by time, not by the update of an attribute. I acknowledge that currently the spec is not very clear about it - so clarifications need to be added. Possibly, we could also decide to extend the feature to explicitly show updated values also for cases where the notification was not triggered by this change. This will require a good cost - benefit analysis.

Martin

[ScorpioBroker]

Hi,

just to add my 2 cents from the technical side. Currently interval subs can't get the previous value at all. they would need to store it in the DB since multiple instances of the sub man need to be aware of a change and on every interval everyone has to pull again from the DB ... i would say normally not a big problem but in a big case with lots of subs or lots of data in the notification and a fast interval you might end up with a delay bigger then your interval ... just for completeness sake... onchange on the other side will be triggered by message which can in all cases include the previous value from the DB directly so there it is always there

[daniel-gonzalez-sanchez]

Thanks again for your quick answer and clarifications!

Following the ETSI CIM standard and the specification document for the NGSI-LD API, it was not clear to me wether the showChanges feature would be valid or not for periodic subscriptions and it seemed that it was something global for any type of subscription (i.e. periodic or change-based). Thanks for the details about the original intention of this feature and about the technical side. I understand the problem of the showChanges feature in periodic subscriptions, however I think that from the point of view of users or consumers there could be practical applications in using this for periodic subscriptions.

Also, note the last comment of the previous message that for notifications on periodic subscriptions that configure the sysAttrs feature, when there is an update to the subscribed entity the modifiedAt system-generated attribute has a datetime with precision of seconds instead of milliseconds. That is, the first notifications resulting from the creation of the entity have a modifiedAt with millisecond precision but as soon as there is an update within the entity the precision is changed to seconds. I have observed this with the sysAttrs feature set and with or without the showChanges feature set.

Thanks again!