Thingsboard Edge Not Deploying Downlinks [Bug]

[apadEnlace]

Describe the bug Thingsboard (TB) Edge not able to Deploy changes/updates from TB Cloud. Updates are labeled as "pending" and they don't get processed:

If I restart the TB Edge service, changes/updates done to rule chain get implemented; I'm not seeing that values that I push to the edge with the node "push to edge" are getting deployed. The whole deployment process seems to have issues.

This issue started after we did quite a few changes in a rule chain.

When I look at the Errors in TB Edge, I see the following:

Any advise would be appreciated!

Your Server Environment OS: Ubuntu 20.04 ThingsBoard: TB Edge PE (3.3.4pe) and TB Server PE 3.3.2 Browser: Chrome Version 100.0.4896.60

Expected behavior Changes/Updates get deployed as soon as they happen on the cloud

[volodymyr-babak]

@apadEnlace hello, if you specified your versions I suppose the issue is because the edge version (3.3.4pe) is higher than your server version (3.3.2). We had stated in the upgrade instructions this:

But probably you had just installed Edge from scratch... We are going to add this to the install guide and additionally, in the next release we'll target to add a check of the version on edge/server sides and stop edge in case versions are not compatible.

Are you able to upgrade your TB server to the latest 3.3.4.1 version?

[apadEnlace]

Hi @volodymyr-babak,

That's correct; I installed TB Edge from scratch. I'll upgrade TB Server and comment my observations

Thanks!

[apadEnlace]

I've upgraded TB server to 3.3.4.1:

I've rebooted the machine hosting TB Edge. I still see the downlinks stuck (in "Pending" status):

Errors in TB Edge:

I'm tempted to re-install TB Edge. Let me know your comments

Thanks

[volodymyr-babak]

@apadEnlace thanks for updating server and checking it. Could you please attach /var/log/thignsboard/thingsboard.log file and /var/log/tb-edge/tb-edge.log files?

[apadEnlace]

Please find the logs attached. Both files are Zipped.

tb-edge.zip thingsboard.zip

[volodymyr-babak]

@apadEnlace

thanks for the logs. I believe the server is still trying to deliver persisted messages for the 3.3.2 version. Please do next steps to unblock it:

1. Change queueStartTs attribute of the edge instance to the current timestamp:
2. Click 'Sync Edge' button after queueStartTs was updated:

[apadEnlace]

Thanks for the reply.

I did the steps. I edited "queueStartTS" and clicked "Sync Edge" (which gave me "successful" message). What I see in the downlinks is that TB keeps adding "Rule Chain Metadata" events (see below). Not sure where these are coming from; might be old changes to the rule chain in the edge instance

Screenshot of a portion of the content in "DATA" of one of the downlinks:

Additionally, as a test, I tried updating the value of a "Shared Attribute" of a device in the edge from TB Server:

However, attribute value on edge instance doesn't update:

This update might be stuck in the sea of downlinks being triggered by the cloud TB server (as seen in first screenshot above).

I attached must recent log files: tb-edge_AND_TB-Server_logs.zip

Thanks for looking into this

[volodymyr-babak]

@apadEnlace could you please set queueStartTs in milliseconds and not in seconds? The value should be 1653375758692 or bigger.

So please do next steps:

1. stop edge
2. update queueStartTs on server of this edge to the current time in milliseconds
3. start edge
4. once it's connected please click 'Sync Edge' button

[apadEnlace]

@volodymyr-babak,

I had issues with the VM where I had TB Edge and ended up rebuilding it and reinstalling TB Edge. I started observing that I was getting the same issues with the downlinks in the new install. At the same time, I noticed that the clock of the VM was off; so, I proceeded to sync it, and TB Edge downlinks have been working fine since then!

I'm positive that the issue with TB Edge in the previous VM was also clock being off

Thanks for your help @volodymyr-babak

[apadEnlace]

Closing issue