[Bug] Device to Server RPC Requests not working via Edge

[thirstyfish]

Describe the bug We're using device to server RPC requests, which works as expected when connecting the device to the TB server directly or via a gateway using the MQTT api. However, when the device is connected to TB Edge, the edge can't relay the RPC call to the server and throws the following error:

java.lang.RuntimeException: Unsupported msg type TO_SERVER_RPC_REQUEST
    at org.thingsboard.rule.engine.edge.AbstractTbMsgPushNode.onMsg(AbstractTbMsgPushNode.java:65)
    at org.thingsboard.server.actors.ruleChain.RuleNodeActorMessageProcessor.onRuleChainToRuleNodeMsg(RuleNodeActorMessageProcessor.java:135)
    at org.thingsboard.server.actors.ruleChain.RuleNodeActor.onRuleChainToRuleNodeMsg(RuleNodeActor.java:102)
    at org.thingsboard.server.actors.ruleChain.RuleNodeActor.doProcess(RuleNodeActor.java:61)
    at org.thingsboard.server.actors.service.ContextAwareActor.process(ContextAwareActor.java:45)
    at org.thingsboard.server.actors.TbActorMailbox.processMailbox(TbActorMailbox.java:142)
    at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Edge rule chain:

The documentation of the Action - push to cloud rule node states that it only accepts the following msg types:

POST_TELEMETRY_REQUEST POST_ATTRIBUTES_REQUEST ATTRIBUTES_UPDATED ATTRIBUTES_DELETED ALARM

, so the TO_SERVER_RPC_REQUEST message type is missing.

Is there a different way to implement this functionality or is this a bug?

Your Server Environment

• own setup
  • TB 3.4 running in k8s
  • TB Edge 3.4 running in Docker on the edge

Your Device

• Connectivity
  • MQTT

To Reproduce Steps to reproduce the behavior:

1. Send device to server RPC request via MQTT directly to TB edge and observe the Push To Cloud rule node throwing the above error.

Expected behavior The device to server RPC request should be relayed to the server regardless of the TB edge being used or the device being connected to the server directly

[volodymyr-babak]

hi @thirstyfish

Could you please provide more information on your use case? Why you would like to send this RPC Request from the device to the cloud and not process it directly on the Edge?

That's is true, push to cloud rule node doesn't support these types of messages. We are going to update the default Edge Root Rule chain and remove Success connection to push to cloud rule node.

But still, please provide more details on your use case, so we can provide you with a solution to your problem.

[thirstyfish]

Hi @volodymyr-babak - thanks for your response.

We're using device to cloud RPCs to estimate uplink latency (RPC roundtrip time) for now.

However, in the future, the following use cases would be introduced:

• calls to 3rd party commercial APIs that are not possible to distribute access keys for to every single edge instance so they need to be called from a single cloud service via an RPC from the edge.
• calls to frequently changing business rules/logic that's outside the scope or impractical to manage in ThingsBoard and is managed in a different cloud service

We could rely on a simple protocol built on top of client side / shared attributes for up/down comms for all the above scenarios but it unnecessarily overcomplicates things and would be inelegant.

It'd also be ideal to be able to expose the same capabilities to edge devices regardless of whether they're connected to a TB edge server or to a local gateway device that's connected to the cloud TB via the MQTT gateway API.

[AndreMaz]

@volodymyr-babak any update on this? I'm also interested in this feature.

Specially in this part:

It'd also be ideal to be able to expose the same capabilities to edge devices regardless of whether they're connected to a TB edge server or to a local gateway device that's connected to the cloud TB via the MQTT gateway API.

[volodymyr-babak]

hi @thirstyfish @AndreMaz

at the moment RPC requests are works only for the local device session. But we consider this as a bug/enhancment, that should be addressed in future releases. We are going to design this feature and provide an update on the expected release of version 3.4.2 or 3.5. The general problem in this feature is round trip from cloud to the device over edge and usage of general downlink queue for all types of messages. Probably we are going to introduce some kind of high-priority queue to achieve RPC and shared attribute subscription from cloud to end device over the edge.

[thirstyfish]

Thanks @volodymyr-babak Just to clarify, could you please confirm that the downstream (cloud => edge => device) RPCs don't work either at the moment, not just the upstream (device => edge => cloud) RPCs which were the original topic for this ticket? Bumped into this downstream issue today and just want to ensure we're not debugging something that's not supposed to work yet. Thanks!

[volodymyr-babak]

@thirstyfish sorry for the delay - somehow I missed your latest comment on this topic. Could you please provide more details on your downstream RPC request? Is it oneway or twoway? Both these RPC requests from the cloud to the device connected to the edge should be working and we have integration tests special for these cases: https://github.com/thingsboard/thingsboard-edge/blob/master/msa/edge-black-box-tests/src/test/java/org/thingsboard/server/msa/edge/DeviceClientTest.java#L462

Is your device assigned to a single edge?

Do you have in the downlink section of the edge any 'Pending' requests?

[thirstyfish]

Thanks for getting back on this @volodymyr-babak

We've put the testing of TB-edge on hold for now, so re-deployed it to retest and the downstream RPCs seem to be working fine now.

It may have been an issue with the device <-> edge assignment as you pointed out, apologies for the false alarm.

Do you have a planned ETA for the upstream RPCs by any chance?

Cheers

[volodymyr-babak]

Hello @thirstyfish pull request was created yesterday https://github.com/thingsboard/thingsboard/pull/7592

I hope to fit this fix into 3.4.2 release, that is coming soon (approx. 2-3 weeks)

[thirstyfish]

That's great news, thanks for the update @volodymyr-babak ! Cheers

[volodymyr-babak]

pull request was merged. fix is already in the master branch and is going to be available next release.