openthread / openthread

OpenThread released by Google is an open-source implementation of the Thread networking protocol
https://openthread.io
BSD 3-Clause "New" or "Revised" License
3.49k stars 1.07k forks source link

[github-actions] stress connectivity fails intermittently #7406

Closed jwhui closed 2 years ago

jwhui commented 2 years ago
Traceback (most recent call last):
#### ML-EID Connectivity Test **Passed** _(02/10 01:26:21 ~ 02/10 01:35:08)_
  File "/tmp/otns/pylibs/stress_tests/run_stress_suite.py", line 95, in <module>
| Simulation Time | Max Delay | Min Delay | Avg Delay |
    main()
  File "/tmp/otns/pylibs/stress_tests/run_stress_suite.py", line 70, in main
    run_suite(script_dir, suite_name)
  File "/tmp/otns/pylibs/stress_tests/run_stress_suite.py", line 91, in run_suite
    t.run(report=True)
  File "/tmp/otns/pylibs/stress_tests/BaseStressTest.py", line 60, in run_wrapper
    self.report()
  File "/tmp/otns/pylibs/stress_tests/BaseStressTest.py", line 120, in report
    raise RuntimeError("Stress test failed: \n" + "\n".join("\t" + msg for msg in self.result._fail_msgs))
RuntimeError: Stress test failed: 
    Max Delay (4800s)> 3600s
| :-: | :-: | :-: | :-: |
| 6h | 2700s | 60s | 645s |
**[OTNS](https://github.com/openthread/ot-ns) Stress Tests Report Generated at 02/10 01:43:30**
#### Service Connectivity Test **Failed**: Max Delay (4800s)> 3600s _(02/10 01:35:08 ~ 02/10 01:43:30)_
| Simulation Time | Max Delay | Min Delay | Avg Delay |
| :-: | :-: | :-: | :-: |
| 6h | 4800s | 60s | 1468s |
Error: Process completed with exit code 1.

logs_113924.zip https://github.com/openthread/openthread/runs/5134548033?check_suite_focus=true

simonlingoogle commented 2 years ago

This, the connectivity stress test has been a little flaky.

This failure shows that some SED device failed to upload data to the BR (by Service ALOC) in 4800 seconds (basically during the whole test), due to 20% packet loss ratio and constantly node moving.

A simple fix would be reducing the packet loss ratio from 20% to 10%, which we already have a PR: https://github.com/openthread/ot-ns/pull/198

I will take a further look to see if https://github.com/openthread/ot-ns/pull/198 is a good way to go.