Edge PE errors related to "change originator" node

[akseerali]

Component

• Rule Engine

Description The TB Edge PE 3.6.3 container throws errors for the change originator node. After checking the Rule Engine Statistics under the API Usage, it was found that most of the Change Originator nodes show "Failed to initialize rule node." Another node, the "Flow Rule Chain", was also found to be affected. Some of the error logs from edge container are attached. tb-edge-originator error-logs.txt

The following steps are tried to clear this issue:

• Restarted containers
• Removed the edge container and then restarted again.
• Updated rule chain configuration from the cloud.
• Removed the rule chains and devices from the cloud, but the edge still shows them.
• Restored the previous stable edge version (it also showed the same error).
• Upgraded tb-edge-pe from 3.6.3 to 3.6.4, but the edge container stops after few seconds.

Further observations:

• When there is an update of rule chain configuration, the edge gets an update after selecting the synchronization option.
• The edge root rule chain is working as it do not consists any Change Originator or Flow Rule Chain node.
• Postgres container is showing below errors:

  postgres-db  | 2024-07-04 19:47:14.474 IST [81] ERROR:  uncommitted xmin 147587832 from before xid cutoff 190340898 needs to be frozen
  postgres-db  | 2024-07-04 19:47:14.474 IST [81] CONTEXT:  automatic vacuum of table "tb-edge.pg_catalog.pg_constraint"
  postgres-db  | 2024-07-04 19:47:14.485 IST [81] ERROR:  uncommitted xmin 167623156 from before xid cutoff 190340898 needs to be frozen
  postgres-db  | 2024-07-04 19:47:14.485 IST [81] CONTEXT:  automatic vacuum of table "tb-edge.public.relation"
  postgres-db  | 2024-07-04 19:47:14.516 IST [81] ERROR:  uncommitted xmin 167635114 from before xid cutoff 190340899 needs to be frozen
  postgres-db  | 2024-07-04 19:47:14.516 IST [81] CONTEXT:  automatic vacuum of table "tb-edge.public.rule_node

Queries:

• Please provide any help to clear this issue.
• First priority is to ensure the safety of database.
• In the second priority, we tried to restore the previous version, but it didn't work. Is there any method to safety revert back to backup without appearance of new log errors in the previous edge version?

Environment

• OS: Ubuntu 22.04
• ThingsBoard Edge: tb-edge-pe:3.6.3EDGEPE
• TB Edge docker compose configuration tb-edge-configuration.txt

---

Disclaimer

We appreciate your contribution whether it is a bug report, feature request, or pull request with improvement (hopefully). Please comply with the Community ethics policy, and do not expect us to answer your requests immediately. Also, do not treat GitHub issues as a support channel.

[AndriiLandiak]

Hi, @akseerali.

Based on the logs, you are using ThingsBoard 3.7.0 and Edge 3.6.3/3.6.4. The ### forwardMsgToDefaultRuleChain field for the Change Originator node was added in 3.7.0. To fix this issue, please upgrade for compatibility.

We are committed to ensuring backward compatibility, but sometimes new features cause issues. The recent refactoring in the Major 3.7.0 release has impacted compatibility with Edge 3.6.3/3.6.4. Upgrading to 3.7.0 will resolve this and bring additional improvements.

[akseerali]

Hi @AndriiLandiak

Thanks for the response.

The issue persists.

• After upgrading from 3.6.3 to 3.6.4, the edge container shows error. Please check the attached logs. logs-after-upgrading-to-3.6.4.txt

• In the next step, we tried to upgrade the edge version to 3.7.0 to clear the issue. But it shows some errors during the upgradation process. Please check the logs obtained while upgrading to 3.7.0. logs-when-upgrading-to-3.7.0.txt

[akseerali]

Hi @AndriiLandiak

Here is an update.

In our previous attempt, we upgraded the edge pe instance from 3.6.3 to 3.6.4; however, the upgrade was unsuccessful when we tried to upgrade the version to 3.7.0 due to the duplicate key value errors. To solve this issue, we have reverted back to 3.6.3 and removed the duplicated key value entries from the postgres database using SQL script. Below are the detailed steps for removing the duplicated key value appearing during the upgrade to 3.7.0, as discussed in the previous logs attached here.

logs-when-upgrading-to-3.7.0.txt

Steps to clear the duplication issue using SQL queries

• Check the duplication of identified key value entries from logs

  SELECT entity_id, attribute_type, attribute_key, COUNT(*)
  FROM attribute_kv
  GROUP BY entity_id, attribute_type, attribute_key
  HAVING COUNT(*) > 1;

• Create a backup using SQL query CREATE TABLE attribute_kv_backup AS TABLE attribute_kv;

• Delete the duplicate record

  DELETE FROM attribute_kv a
  WHERE ctid NOT IN (
  SELECT max(ctid)
  FROM attribute_kv b
  WHERE a.entity_id = b.entity_id
    AND a.attribute_type = b.attribute_type
    AND a.attribute_key = b.attribute_key
  );

Duplication of email issue The logs in Edge version 3.6.3 also showed the duplicated key value error for an email address assigned to Edge from the cloud. We simply removed the user from the cloud to clear this error, which solved the issue.

Upgradation of edge version to 3.7.0 After resolving the issue of duplicated key value errors, we were able to successfully upgrade the edge from 3.6.4 to 3.7.0 this time. Here is file containing logs obtained during the upgradation of edge from 3.6.4 to 3.7.0. logs-when-upgrading-to-3.7.0-v2.txt

After upgrading the edge, we started the edge instance. The edge container stopped after a few seconds, indicating the software version does not support this license. Please find below the logs from the edge container after running the first time post-upgrade. logs-after-upgrading-to-3.7.0.txt

Queries

• It's been more than one year since we purchased our perpetual edge software license. As per the information from the documentation (Link), we might need to get another license for the edge to upgrade. Please confirm if you need to get another license.
• If the license is not the problem, we can revert to either the previous edge backup or even the changes made in the Postgres database. Please let us know about the feedback.
• Please let us know if there are any other troubleshooting steps. With edges 3.6.3 and 3.6.4, the system shows the errors identified in the first comment.

  Description The TB Edge PE 3.6.3 container throws errors for the change originator node. After checking the Rule Engine Statistics under the API Usage, it was found that most of the Change Originator nodes show "Failed to initialize rule node." Another node, the "Flow Rule Chain", was also found to be affected. Some of the error logs from edge container are attached. tb-edge-originator error-logs.txt

[AndriiLandiak]

Hi, @akseerali!

Great job on resolving the issue with the attribute_kv constraints.

Based on the logs, there's a problem regarding the perpetual rule of "1 year of software updates."

Could you please create a ticket on the customer portal with your information, including the license ID for your current subscription? Alternatively, you can email me at alandiak@thingsboard.io. We will generate a new license for you to upgrade the Edge to version 3.7.0.

If you would like to continue receiving new updates, you might consider purchasing a new perpetual subscription for the entire year.

[akseerali]

The issue is resolved after upgrading the edge pe version from 3.6.3 to 3.7.0.