Closed reubenmiller closed 11 months ago
QA has thoroughly checked the feature and here are the results:
Detailed results: The test script provided in the issue was run in the new version and no errors detected:
sudo ./debug-qos-issue.sh
Max number of runs reached (disabling service)
20 passed, 0 failed, 20 total
And no errors detected in the tedge-mapper-c8y logs either:
$ journalctl -u tedge-mapper-c8y -n 10000 | grep JWT
Nov 11 09:21:59 bbf0c2a0fe36 tedge-mapper[783]: 2023-11-11T09:21:59.939622358Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:22:00 bbf0c2a0fe36 tedge-mapper[783]: 2023-11-11T09:22:00.322978399Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:22:08 bbf0c2a0fe36 tedge-mapper[1031]: 2023-11-11T09:22:08.012651861Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:22:08 bbf0c2a0fe36 tedge-mapper[1031]: 2023-11-11T09:22:08.03971257Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:24:36 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:24:36.416902099Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:24:36 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:24:36.787357474Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:24:54 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:24:54.205046191Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:24:54 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:24:54.659474524Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:25:11 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:25:11.849651657Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:25:12 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:25:12.765703741Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:25:29 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:25:29.547639708Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:25:30 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:25:30.153399249Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:25:47 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:25:47.356950258Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:25:47 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:25:47.489894216Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:26:05 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:26:05.280921683Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:26:06 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:26:06.356814725Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:26:22 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:26:22.76605565Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:26:23 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:26:23.177114358Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:26:40 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:26:40.448079741Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:26:40 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:26:40.978993283Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:28:28 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:28:28.528151876Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:28:29 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:28:29.129787001Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:28:46 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:28:46.191898593Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:28:47 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:28:47.013378385Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:29:04 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:29:04.114964477Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:29:04 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:29:04.660293602Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:29:22 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:29:22.229899568Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:29:23 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:29:23.120602486Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:29:39 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:29:39.886122077Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:29:40 bbf0c2a0fe36 tedge-mapper[62]: 2023-11-11T09:29:40.44406816Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:29:57 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:29:57.781517835Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:29:58 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:29:58.485614586Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:30:15 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:30:15.812048177Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:30:16 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:30:16.544919053Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:30:33 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:30:33.660222019Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:30:34 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:30:34.115250686Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:30:51 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:30:51.370409194Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:30:51 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:30:51.99424982Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:31:09 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:31:09.039822453Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:31:09 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:31:09.65310262Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:31:26 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:31:26.902802545Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:31:27 bbf0c2a0fe36 tedge-mapper[61]: 2023-11-11T09:31:27.40605692Z INFO c8y_api::http_proxy: JWT token received
Nov 11 09:31:44 bbf0c2a0fe36 tedge-mapper[78]: 2023-11-11T09:31:44.641433428Z INFO c8y_api::http_proxy: JWT token requested
Nov 11 09:31:45 bbf0c2a0fe36 tedge-mapper[78]: 2023-11-11T09:31:45.33213272Z INFO c8y_api::http_proxy: JWT token received
Describe the bug
Strange behaviour of the tedge-mapper-c8y leads to sporadic initialization failure where it is blocked waiting for a jwt (token) from Cumulocity IoT. The behaviour was observed in the tedge-demo-container running tedge 0.13.0.
When the main device container is starting up for the first time, the tedge-mapper-c8y sometimes gets stuck in the initialization because it is unable to successfully request a jwt from Cumulocity. The mapper continuously repeats trying to request a token and is continuously unsuccessful. This blocks all of the tedge-mapper-c8y function.
After doing some debugging it was found that when the tedge-mapper-c8y is stuck in the initialization stage, a manual jwt can still be requested as along as the QoS is set to 0 (and not 1 which is what the tedge-mapper-c8y is using).
Some additional observations:
c8y/s/uat
with QoS 1 (QoS 0 works every time)A test script was written to manually check if requesting a token works or not after a reboot (see "to reproduce" section for details). The following are the results, which show that there seems to be some unexplained negative side-effects when the tedge-mapper-c8y is running and using qos 1.
Case 1: With tedge-mapper-c8y enabled and publishing with QoS 1
Case 2: With tedge-mapper-c8y disabled and publishing with QoS 1
Case 3: With tedge-mapper-c8y enabled and publishing with QoS 0
To Reproduce
Note: The error state (not receiving a token response) only occurs sporadically, these instructions are used to highlight how to identify when it is in the error state. For best results use the tedge-demo-container for testing this scenario:
It is assumed that the following is satisfied before:
Open a new console and subscribe to the Cumulocity topic
Open a different console and request a token using QoS 1
Observe in the first console that a token is not received on the expected
c8y/s/dat
topic:In the same console, publish the same token request message but with Qos 0
Observe in the subscription console that a token is now received!
To help with debugging a script was created, though your mileage might vary as it only has been tested in the demo container. It will need modification if it is to run on a real device in order to wait until the "system is up" before running the test.
Note: Please review the script yourself before trying it out as it needs sudo rights to trigger a restart and install a service.
Expected behavior
The tedge-mapper-c8y should be able to reliably request the jwt from Cumulocity.
Screenshots
tedge-mapper-c8y logs
These logs show how the error manifests in the tedge-mapper-c8y:
mosquitto
The mosquitto logs from the same time period when the agent is in the initialization loop waiting for a token.
Environment (please complete the following information):
Debian GNU/Linux 11 (bullseye)
unknown
Linux 17a2666b257d 5.15.68-0-virt #1-Alpine SMP Fri, 16 Sep 2022 06:29:31 +0000 aarch64 GNU/Linux
tedge 0.13.0
2.0.11
Additional context