The bridge to Cumulocity doesn't subscribe to sub topics under `s/dc/#`

[didier-wenzek]

Describe the bug

On tedge connect c8y thin-edge properly configures the mosquitto bridge with topic s/dc/# in 2 c8y/ "" to forward operation requests from the cloud to the device. This works fine for pre-defined operations, but if one creates, on Cumulocity, a new SmartRest operation template that publish operations on a sub-topic, say s/dc/c8y-dm-agent-v3.0 then these custom operations are not received on the corresponding s/dc/c8y-dm-agent-v3.0 device topic, unless this sub-topic is explicitly added to the bridge configuration and mosquitto restarted.

To Reproduce Citing @mbay-SAG:

I found some strange behavior in thin-edge regarding smartrest templates which I can not explain. @artur.toews pointed me to that. I uploaded a SmartRest Template. If I subscribe to s/dc/# i do receive the operation if created on that particular device if I do it outside thin-edge but within e.g. MQTT.fx (see screenshot 1). However if I do the same on thin-edge I do not get the same operation on s/dc/# but only if i change the c8y-bridge.conf file to subscribe directly on that particular smartrest ID. (See screenshot 2). Can you double check that? I do not see why s/dc/# is not working properly. I only have the Issue on thin-edge.

Expected behavior

Since the bridge configuration explicitly subscribe to all the sub-topics s/dc/#, the custom operations should be received without the need to configure explicitly the bridge.

Screenshots

Environment (please complete the following information):

Additional context

This remember me the issue we already observe with operations when creating the bridge too soon. At that time the fix has been to restart mosquitto twice. The main reason was related to the fact that the MQTT end-point of Cumulocity is not a full-fledged MQTT server and seems to not correctly handle topics created on the fly.

To double check that one needs to reproduce the issue in two different settings:

1. using Cumulocity
2. using a remote MQTT server that is not Cumulocity but from which we can send operation requests.

[mbay-ODW]

Any feedback on this? I know have a setup where we can debug this. I can reproduce the issue.

[albinsuresh]

It doesn't look like a thin-edge specific issue. I could reproduce the issue without thin-edge or mosquitto bridge in the picture. I connected directly to Cumulocity using the mosquitto_sub client with client certificates and this issue is reproducible with that direct connection as well.

Subscriptions to s/dc/# don't receive the operations sent using SmartREST template, as follows:

$ sudo mosquitto_sub -d -h thin-edge-io.eu-latest.cumulocity.com -p 8883 --capath /etc/ssl/certs --key /etc/tedge/device-certs/tedge-private-key.pem --cert /etc/tedge/device-certs/tedge-certificate.pem -i albin -t s/dc/#
Client albin sending CONNECT
Client albin received CONNACK (0)
Client albin sending SUBSCRIBE (Mid: 1, Topic: s/dc/#, QoS: 0, Options: 0x00)
Client albin received SUBACK
Subscribed (mid: 1): 0

As you can see, no messages received after the subscription.

But when you connect an MQTT client with username and password, subscriptions to both s/dc/# and s/dc/<template-id> receive the operation messages without any issues.

It looks like this issue happens only with clients connecting to Cumulocity using certificates. My test setup above and thin-edge mosquitto bridge are both examples of those. So, we might have to raise this issue with Cumulocity.

[albinsuresh]

Cumulocity team has confirmed that subscriptions to s/dc/# is not supported by them and they're not planning to support it either. With such a wildcard subscription to "everything", the device will get "everything from every template" delivered to it as well, even though device was only interested in responses from one template. It can create a lot of unintended side effects to devices, consume a lot of compute power on the server and even bandwidth sending un-necessary data to all thin-edge devices subscribed to that topic all the time.

So, we will have to work out a mechanism to detect the selected set of custom templates that a thin-edge device is interested via configurations or bootstrap messages etc. In short, this will have to be designed as a feature and not treated as a bug. We were trying to use something that Cumulocity never supported.

[mbay-ODW]

Any comment why this works if connected via username/pw but not if connected via Device Certificate? In my opinion it does not make sense to threat this differently.

However that statement heavily impacts the design of thin-edge.io, since custom smarrest templates do not work with that approach.

[albinsuresh]

But when you connect an MQTT client with username and password, subscriptions to both s/dc/# and s/dc/<template-id> receive the operation messages without any issues.

That was a bad observation from my end. I had a client that had subscribed to both s/dc/# and s/dc/<template-id> simultaneously and then, when a message was delivered to the handler of s/dc/<template-id>, the same message was delivered to the handler of s/dc/# as well, giving me a false sense of that subscription working. If the client subscribes to s/dc/# only, then the messages don't even reach from the cloud. In my case, the messages got delivered just because there was a selective subscription to s/dc/<template-id> as well.

However that statement heavily impacts the design of thin-edge.io, since custom smarrest templates do not work with that approach.

Totally agree and yes we need to find a solution to that problem without expecting a s/dc/# like contract from Cumulocity which will never happen. We'll need a to support a mechanism that allows users to declare that they're gonna use some custom templates and then thin-edge selectively subscribing to those. This "declaration of interested templates" can be configuration based or dynamic discovery based or trigger based (when the user uses a template for the first time). But, all of that will have to be decided as part of the grooming of this custom template support "feature". There is no quick workaround that I can offer at the moment.

[didier-wenzek]

Cumulocity team has confirmed that subscriptions to s/dc/# is not supported by them and they're not planning to support it either. With such a wildcard subscription to "everything", the device will get "everything from every template" delivered to it as well, even though device was only interested in responses from one template. It can create a lot of unintended side effects to devices, consume a lot of compute power on the server and even bandwidth sending un-necessary data to all thin-edge devices subscribed to that topic all the time.

This sounds a bit weird since Cumulocity side s/dc is a device specific topic (a device receives only his operation despite all the devices are subscribing to the same topic). From a logical perspective I understand this device specific topic akin to <device-id>/s/dc.

But, if this is the behavior of Cumulocity, we have do deal with it.

However that statement heavily impacts the design of thin-edge.io, since custom smarrest templates do not work with that approach.

Yes, we need to fix the bridge configuration for Cumulocity and provide a way to explicitly declare new topics to subscribe to.

• There is not point to subscribe to s/dc/#.
• The bridge needs to subscribe to s/dc plus a list of topics s/dc/<NAME> for a list of configured NAMEs.

@albinsuresh & @mbay-ODW, I let you find the best way to configure this list of specific topics and generate the appropriate bridge configuration.

[albinsuresh]

@mbay-ODW We're thinking of providing a way for users to declare the interested templates via a tedge configuration like tedge config set c8y.smartrest.templates template-1,template-2 which can be set before the user performs initial tedge connect c8y. If he wants to add more templates in future, he'll have to do tedge disconnect c8y first, run tedge tedge config set c8y.smartrest.templates template-1,template-2,new-template-3 and then tedge connect c8y again.

Does that UX work for you?

[mbay-ODW]

As discussed today the mentioned approach looks good. However there might be a change in smartRest implementation that would solve this in long term.

[mbay-ODW]

@albinsuresh Double checked about customer expectation as well. For devices without child devices the above mentioned approached works fine. However in the context of child devices this might not work since parent devices do not know which templates a child device would be able to use.

My personal suggestion would thus be start with the above mentioned approach and lets have an discussion about how this could be handled for parent devices with child devices.

[albinsuresh]

This feature will be available in the upcoming 0.7 release. Documentation here: https://thin-edge.github.io/thin-edge.io/html/howto-guides/024_smartrest_templates.html