boards/microbit: tracker for CI blacklist due to test failing with qemu

kaspar030 commented 2 years ago

Description

https://github.com/RIOT-OS/RIOT/pull/17434 enables CI to test for microbit emulated with qemu. Some tests fail. To get it in, they have been CI blacklisted for further investigation.

IMO, they all need to be checked if the failure is related to qemu not emulating hardware, or qemu not being exact enough (timing off?), or something wrong with the setup (pyterm seems to make some tests fail?). So I dump the list here.

As a reminder, if qemu is installed (qemu-system-arm needs to be available), a test can be run like this:

EMULATE=1 BOARD=microbit make -Ctests/fmt_print all test

Missing rtt:

[x] tests/periph_rtt_min

Not exact enough:

[x] tests/ztimer_overhead -> timing off
[x] tests/thread_flags -> timing off

Unknown:

[ ] tests/evtimer_mbox
[ ] tests/driver_nrfmin
[ ] tests/gnrc_ipv6_fwd_w_sub
[ ] tests/heap_cmd
[ ] tests/evtimer_msg
[ ] tests/netstats_l2
[ ] tests/event_wait_timeout_ztimer
[ ] tests/mtd_raw
[ ] tests/evtimer_underflow
[ ] tests/periph_gpio
[ ] tests/gnrc_netif_ipv6_wait_for_global_address
[ ] tests/gnrc_sixlowpan_frag_sfr
[ ] tests/periph_flashpage
[ ] tests/bench_xtimer_load
[ ] tests/gnrc_tx_sync
[ ] tests/periph_wdt
[ ] tests/log_color -> looks like a term issue
[ ] tests/lwip_sock_udp
[ ] tests/test_tools
[ ] tests/struct_tm_utility
[ ] tests/ps_schedstatistics
[ ] tests/sys_atomic_utils
[ ] tests/pthread_cooperation
[ ] tests/xtimer_usleep
[ ] tests/lwip_sock_ip
[ ] tests/turo_txt
[ ] tests/xtimer_now64_continuity
[ ] tests/sys_sched_round_robin
[ ] tests/xtimer_rmutex_lock_timeout
[ ] tests/xtimer_periodic_wakeup
[ ] tests/ztimer_rmutex_lock_timeout
[ ] tests/ztimer_periodic
[ ] tests/periph_rtt
[ ] tests/trickle
[ ] tests/posix_semaphore
[ ] tests/xtimer_mutex_lock_timeout
[ ] tests/xtimer_hang
[ ] tests/shell
[ ] tests/xtimer_usleep_short
[ ] tests/ztimer_underflow
[ ] tests/posix_sleep
[ ] tests/ztimer_xsec
[ ] tests/lwip_sock_tcp
[ ] tests/pkg_umorse

Steps to reproduce the issue

Expected results

Actual results

Versions

kfessel commented 2 years ago

seems like it is not unlikely that these emulates tests fail due to timing issues

(I ran the tests at my laptop and they succeeded)

``` -- running on worker comsys thread 4, build number 17514. make: Entering directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout' Building application "tests_event_wait_timeout" for "microbit" with MCU "nrf51". sha1sum /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout/tests/01-run.py /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/tests_event_wait_timeout.elf > /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/test-input-hash.sha1 text data bss dec hex filename 12372 176 3428 15976 3e68 /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/tests_event_wait_timeout.elf make: Leaving directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout' make: Entering directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout' r /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/dist/tools/emulator/term.sh qemu microbit /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout /tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/dist/tools/pyterm/pyterm '-p "/tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart" -b "115200" ' /tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart /tmp/riot_tests_event_wait_timeout_microbit.jTfrf qemu-system-arm -machine microbit -device loader,file=/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/build/tests_event_wait_timeout.elf -serial unix:/tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart_socket,server=on,wait=off -monitor unix:/tmp/riot_tests_event_wait_timeout_microbit.jTfrf/mon_socket,server=on,wait=off -nographic Twisted not available, please install it if you want to use pyterm's JSON capabilities 2022-04-11 17:21:10,326 # Connect to serial port /tmp/riot_tests_event_wait_timeout_microbit.jTfrf/uart Welcome to pyterm! Type '/exit' to exit. 2022-04-11 17:21:11,329 # READY s 2022-04-11 17:21:11,382 # START 2022-04-11 17:21:11,383 # main(): This is RIOT! (Version: buildtest) 2022-04-11 17:21:11,384 # [START] event_wait_timeout test application. 2022-04-11 17:21:11,384 # Timeout in expect script at "child.expect_exact(u"[SUCCESS]")" (tests/event_wait_timeout/tests/01-run.py:15) make: *** [/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/makefiles/tests/tests.inc.mk:22: test] Error 1 make: Leaving directory '/tmp/dwq.0.44999647515907937/49bc3d5165124b447ca08da986c00022/tests/event_wait_timeout' {"build/": 940} ```

``` -- running on worker mobi8.inet.haw-hamburg.de thread 16, build number 9937. make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' Building application "tests_cb_mux_bench" for "microbit" with MCU "nrf51". sha1sum /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench/tests/01-run.py /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf > /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/test-input-hash.sha1 text data bss dec hex filename 11216 168 2800 14184 3768 /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' Building application "tests_cb_mux_bench" for "microbit" with MCU "nrf51". sha1sum /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench/tests/01-run.py /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf > /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/test-input-hash.sha1 text data bss dec hex filename 11216 168 2800 14184 3768 /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' make: Entering directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' r /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/dist/tools/emulator/term.sh qemu microbit /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench /tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/dist/tools/pyterm/pyterm '-p "/tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart" -b "115200" ' /tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart /tmp/riot_tests_cb_mux_bench_microbit.J0ak2 qemu-system-arm -machine microbit -device loader,file=/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/build/tests_cb_mux_bench.elf -serial unix:/tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart_socket,server=on,wait=off -monitor unix:/tmp/riot_tests_cb_mux_bench_microbit.J0ak2/mon_socket,server=on,wait=off -nographic Twisted not available, please install it if you want to use pyterm's JSON capabilities 2022-04-11 17:08:53,033 # Connect to serial port /tmp/riot_tests_cb_mux_bench_microbit.J0ak2/uart Welcome to pyterm! Type '/exit' to exit. 2022-04-11 17:08:54,042 # READY s 2022-04-11 17:08:54,094 # START 2022-04-11 17:08:54,096 # main(): This is RIOT! (Version: buildtest) 2022-04-11 17:08:54,097 # cb_mux benchmark application 2022-04-11 17:08:54,098 # Populating cb_mux list with 20 items 2022-04-11 17:08:54,099 # Finding the last list entry 2022-04-11 17:08:54,100 # List walk time: 274 us 2022-04-11 17:08:54,101 # Walk time greater than threshold of 200 us 2022-04-11 17:08:54,102 # [FAILURE] 2022-04-11 17:08:54,104 # { "threads": [{ "name": "main", "stack_size": 1536, "stack_used": 436 }]} Timeout in expect script at "child.expect(r"Walk time less than threshold of \d+ us")" (tests/cb_mux_bench/tests/01-run.py:18) make: *** [/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/makefiles/tests/tests.inc.mk:22: test] Error 1 make: Leaving directory '/tmp/dwq.0.6968863166384458/88759950be032501161824f484e892bc/tests/cb_mux_bench' {"build/": 920} ```

kaspar030 commented 2 years ago

yes, very likely. do you want to open a blacklist PR?

kfessel commented 2 years ago

I am not sure since it does not seem happen often (only one other PR shows this atm "PR #16771 build at 11/04/2022, 13:27:52") (maybe this happens when the building server has to much other load?) On the other hand ci.riot / murdock shows only the latest run of a PR. Maybe you have a better view at the past failing builds? (could we ignore the results but still run the test?)

Some of these timeouts might be solved by setting RIOT_TEST_TIMEOUT for emulated microbit

kfessel commented 2 years ago

https://ci.riot-os.org/RIOT-OS/RIOT/17096/c7638d84ae5a709d6634ffe81dbce6828ecbab3f/output/compile/tests/pbkdf2/microbit:gnu.txt https://ci.riot-os.org/RIOT-OS/RIOT/16771/5a6b007467d22bee4ec7278782d272d8df7cd4d5/output/compile/tests/event_wait_timeout/microbit:gnu.txt https://ci.riot-os.org/RIOT-OS/RIOT/16771/5a6b007467d22bee4ec7278782d272d8df7cd4d5/output/compile/tests/pbkdf2/microbit:gnu.txt

kfessel commented 2 years ago

https://qemu.readthedocs.io/en/latest/system/arm/nrf.html

maybe we should make the emulated microbit a seperate board (with a different feature set)

RIOT-OS / RIOT