spec11: harden testing - Githubissues

miri64 commented 2 years ago

This introduces several changes to the LoRaWAN tests, in the hopes to not make them fail that regular anymore:

Parametrize by DR (and the resulting tx_delay)
~~Add some grace time to the waiting between sending from the node and receiving at the TTN client~~
Mark task02 and task03 as flaky. Those regularly fail in the weekly builds. Marking them flaky will try to repeat them at least 3 in case of failure
~~Increase expect timeout on lorawan_tx for task03~~
Retry sending itself a few times for ABP (sometimes it just fails on the first time)
Remove ul_cnt setting. When set, TX for some reason never succeeds.

miri64 commented 2 years ago

There seems to be (appearently unrelated) issue where task 3 can't complete since the lorawan_tx fails. TTN does not show any packets for that, @jia200x is currently trying to figure out what went wrong.

miri64 commented 2 years ago

It appears, the only reason https://github.com/RIOT-OS/RIOT/runs/7673103443?check_suite_focus=true fails now is because the experiment is suddenly stopped while the device tries to send. @aabadie @fjmolinas do you know if that is a known problem?

miri64 commented 2 years ago

From the discussion in https://github.com/RIOT-OS/Release-Specs/issues/258#issuecomment-1227480862, I was thinking: maybe it makes sense to additionally add a blacklist feature for nodes. I.e. if the test fails, add the capability to mark the failing node as defect, and rerun with another. I will investigate this.

miri64 commented 1 year ago

Squashed.

RIOT-OS / Release-Specs

spec11: harden testing #255