Closed miri64 closed 4 years ago
Currently running the tests provided by https://github.com/RIOT-OS/Release-Specs/pull/155, but not putting my name behind them because other people should try it too :-).
First batch of results are in
Failures
03-single-hop-ipv6-icmp.test_spec03.test_task05[nodes0]
04-single-hop-6lowpan-icmp.test_spec04.test_task07[nodes0]
04-single-hop-6lowpan-icmp.test_spec04.test_task08[nodes0]
04-single-hop-6lowpan-icmp.test_spec04.test_task09[nodes0]
Skipped
03-single-hop-ipv6-icmp.test_spec03test_task04[nodes0]
04-single-hop-6lowpan-icmp.test_spec04test_task05[nodes0]
04-single-hop-6lowpan-icmp.test_spec04test_task06[nodes0]
All Test Classes
03-single-hop-ipv6-icmp.test_spec03
test_task01[nodes0]
test_task02[nodes0]
test_task03[nodes0]
test_task04[nodes0]
test_task05[nodes0]
test_task06[nodes0]
04-single-hop-6lowpan-icmp.test_spec04
test_task01[nodes0]
test_task02[nodes0]
test_task03[nodes0]
test_task04[nodes0]
test_task05[nodes0]
test_task06[nodes0]
test_task07[nodes0]
test_task08[nodes0]
test_task09[nodes0]
test_task10[nodes0]
05-single-hop-route.test_spec05
test_task01[nodes0]
test_task02[nodes0]
test_task03[nodes0]
test_task04[nodes0]
06-single-hop-udp.test_spec06
test_task01[nodes0]
test_task02[nodes0]
test_task03[nodes0]
test_task04[nodes0]
test_task05[nodes0]
test_task06[nodes0]
Click on the link, to know more (e.g. why tests were skipped). So a good first run IMHO. Sadly, the stdout
is not provided for the error cases, so I do it here:
03-single-hop-ipv6-icmp.test_spec03.test_task05[nodes0]
This looks like a bug. It might be https://github.com/RIOT-OS/RIOT/issues/12565, but from the log it looks like its on the pinged node, but maybe the output of the test framework is misleading there. Will investigate this.
04-single-hop-6lowpan-icmp.test_spec04.test_task07[nodes0]
There was an error in the deployment of the arduino-zero
. AFAIK @aabadie knows about this issue.
04-single-hop-6lowpan-icmp.test_spec04.test_task08[nodes0]
Same with 04-single-hop-6lowpan-icmp.test_spec04.test_task07[nodes0]
: The testbed did not deploy the arduino-zero
correctly.
04-single-hop-6lowpan-icmp.test_spec04.test_task09[nodes0]
Seems to have been failed due to an error with asyncio
, so probably a bug that needs fixing in #155. Will check what is going on there.
This looks like a bug. It might be RIOT-OS/RIOT#12565, but from the log it looks like its on the pinged node, but maybe the output of the test framework is misleading there. Will investigate this.
No it is not. After having a look with dist/tools/pktbuf-stats/pktbuf-stats.py
I can definitely say that those are ICMPv6 echo responses (type 129) in the buffer, but the nh
of their respective IPv6 header set to PROTNUM_RESERVED
and the link-layer addresses unset in the NETIF header, meaning they are on the pinged node somewhere between allocation and the header being filled up correcty here. Will dig deeper during the week.
dist/tools/pktbuf-stats/pktbuf-stats.py
At least 4.9 now runs successfully, due to my fix in https://github.com/RIOT-OS/Release-Specs/pull/155/commits/9e23b8123054f1db85e8e21f14c99c69486c06dd.
I've also ran the skipped 03-single-hop-ipv6-icmp.test_spec03test_task04[nodes0]
in https://github.com/RIOT-OS/Release-Specs/issues/156#issuecomment-657118379 that requires root permissions:
sudo RIOTBASE=${RIOTBASE} tox -- -m sudo_only
(stdout
is now captured to the results XML :-))
I implicitly ran 07-multi-hop
while testting this branch. Will provide results XML and PR soon.
As always 7.5 is failing (see https://github.com/RIOT-OS/Release-Specs/issues/142#issuecomment-561677974) but the framework accounts for that now.
08-interop Task 11 - UDP exchange between iotlab-m3 and Zephyr: PASS
Edit: I used Zephyr v2.3.0 to test this.
I've automated what I can of 8, and tasks 1 and 8 are succeeding. (the rest is more complicated to automate)
11-lorawan
08-interop:
The testing procedures need to be adapted for https://github.com/RIOT-OS/RIOT/pull/13568 though.
The testing procedures need to be adapted for RIOT-OS/RIOT#13568 though.
I've automated the tests for 10. They require https://github.com/RIOT-OS/RIOT/pull/14510, but they PASS
The CoAP tests are now also fully automated (#168). Here are the passing results
@cgundogan @jia200x @MrKevinWeiss @leandrolanzieri @PeterKietzmann can one of you do 4.5-8, please. The remote
is not in the testbed, so the automated tests don't work there (but you can try if the --local
option of the framework works) and the arduino-zero
s don't want to talk to me (the deployment fails reliably).
02-tests, Task 01
Unable to reproduce locally. I guess, since I am running currently 01-ci
on the same machine at the moment, the scheduler of Linux was just not able to keep up.
however, I can reproduce.
* [tests/xtimer_now_irq](https://github.com/miri64/RIOT-2020.07-results/blob/master/2020.07-RC1/native/tests/)
however, I can reproduce.
From the test's README
This test must be run over at least one full timer period. Meaning the test only finishes in reasonable short time on platforms with small timers up to 24 bit. This test does not work for 32 bit timers.
So that is the reason why it does not work on native
. @JulianHolzwarth why not blacklist (not just CI blacklist) this test for the platforms in question?
I ran all the tests that require root permissions for native
with this command:
sudo dist/tools/tapsetup/tapsetup
for test in tests/{gnrc_dhcpv6,gnrc_ipv6_ext,gnrc_ipv6_nib_dns,gnrc_rpl_srh,gnrc_tcp}*; do
RIOT_CI_BUILD=1 make --no-print-directory -C ${test} -j16 flash &&
sudo make --no-print-directory -C ${test} test ||
break
done
It succeeded
As such, considering https://github.com/RIOT-OS/Release-Specs/issues/156#issuecomment-658457831 and https://github.com/RIOT-OS/Release-Specs/issues/156#issuecomment-658459437, I consider 2.1 to PASS.
iotlab-m3
. Am I right @fjmolinas? If yes, it should be blacklisted for non LoRa-boards.examples/suit_update Made it work locally, but it required some upfront work:
wget https://raw.githubusercontent.com/RIOT-OS/riotdocker/master/requirements.txt
virtualenv -p python3 testenv
. testenv/bin/activate
pip install numpy # required to be able to install emlearn...
pip install -r requirements.txt
pip install --upgrade pexpect aiocoap[linkheader]==0.4b3
sudo dist/tools/ethos/setup_network.sh riot0 2001:db8::/64 ${USER}
In a second terminal
BOARD=iotlab-m3 BUILD_IN_DOCKER=1 MAKE_CMD_ARGS="-j8" make -C examples/suit_update/ flash
sudo BOARD=iotlab-m3 BUILD_IN_DOCKER=1 MAKE_CMD_ARGS="-j8" make -C examples/suit_update/ test
iotlab-m3
native
on the CI. So expected behavior?examples/lorawan
. See https://github.com/RIOT-OS/RIOT/issues/14520git bisect
. See https://github.com/RIOT-OS/RIOT/issues/14521.I seem to have run an old version of RIOT in https://github.com/RIOT-OS/Release-Specs/issues/156#issuecomment-658466453. See https://github.com/RIOT-OS/RIOT/pull/14524.
I as the the error space for spec 02, task 03 overlaps with spec 02, task 02 I don't expect any new insights in trying them locally, and just refer to the latter.
No big surprises in the remaining interop tests.
So only Task #01 - Compile test
missing now (which is running for almost the whole week now... I think we are at a point where that task is not scalable anymore and we should think about just trusting the CI).
And of course 99
. @fjmolinas do you have some results from your paper CI?
So only
Task #01 - Compile test
missing now (which is running for almost the whole week now... I think we are at a point where that task is not scalable anymore and we should think about just trusting the CI).
(on the other hand I think I might have configured something wrong for git-cache
in a docker
. The pkg
tests take suspiciously long...)
I don't have an arduino-zero so I could not run Task 07 and Task 08, but the other 2 look good.
04-single-hop-6lowpan-icmp
And of course
99
. @fjmolinas do you have some results from your paper CI?
I started running them now.
04-single-hop-6lowpan-icmp
Ran using the release test scripts.
I think I got closer to the issue in 3.5: The NIB seems to get stuck in the condition that the next scheduled NS is closer to the next retransmission, so sending the NS is just skipped, causing the retransmission timer not to go down, causing the Neighbor Cache entry to stay, causing the packet to stay in its queue. Will investigate, why it gets stuck.
And of course
99
. @fjmolinas do you have some results from your paper CI?I started running them now.
Talked offline with @fjmolinas: We won't wait for that for RC2. For all currently running there were no surprises. So only thing waiting for RC2 is 01-compiling
(7 applications remaining to be build).
There seems to be a bug in the compile test script. It says 0 errors but there are errors reported:
Have to investigate what is the problem with the failing ones
What I executed was btw
DOCKER_MAKE_ARGS="-j16" \
DOCKER_ENV_VARS=$(env | grep "^CCACHE" | cut -d= -f1) \
DOCKER_VOLUMES_AND_ENV="-v /tmp:/tmp" \
BUILD_IN_DOCKER=1 \
./dist/tools/compile_test/compile_test.py
On deleted comment: tested something ;-).
Ok, most failures are just msp430
nodes failing the linking step (why they are linked and others not and also in other cases not, I don't know, but I assume its rather a bug in the script) but there some other boards failing for riotboot_flashwrite
:
make: Entering directory '/home/mlenders/Repositories/RIOT-OS/RIOT.ssh/tests/riotboot_flashwrite'
Building for slstk3402a ... failed!
Building for sltb001a ... failed!
Building for slwstk6000b-slwrb4150a ... failed!
I can't reproduce this however isolated, by building those boards just with make
, so I assume that for this instance the gecko_sdk
packet was not downloadable at that moment.
Closing in favor of #172
This issue lists the status of all tests for the Release Candidate 1 of the 2020.07 release.
Specs tested:
Task #05 (Experimental) - UDP with large payload on iotlab-m3 with three hops (RPL route)(see https://github.com/RIOT-OS/Release-Specs/issues/156#issuecomment-657259675)Task #09 - UDP between GNRC and emb6 on iotlab-m3(deprecated, see https://github.com/RIOT-OS/Release-Specs/pull/157)Task #10 - UDP between lwIP and emb6 on iotlab-m3(deprecated, see https://github.com/RIOT-OS/Release-Specs/pull/157)