Release 2019.01 - RC1

[aabadie]

This issue lists the status of all tests for the Release Candidate 1 of the 2019.01 release.

Specs tested:

• [x] 01-ci
  • [x] Task #01 - Compile test (@miri64 on Ubuntu 16.04)
  • [x] Task #02 - Unittests on native
  • [x] Task #03 - Unittests on native separated (@aabadie)
  • [x] Task #04 - Unittests on iotlab-m3 (@aabadie)
• [x] 02-tests (@aabadie)
  • [x] Task #01 - All tests on native
  • [x] Task #02 - Subset of tests on iotlab-m3
  • [x] Task #03 - Subset of tests on samr21-xpro
• [x] 03-single-hop-ipv6-icmp (@aabadie)
  • [x] Task #01 - ICMPv6 multicast echo on native
  • [x] Task #02 - ICMPv6 link-local echo on native
  • [x] Task #03 - ICMPv6 link-local echo on native (1 hour)
• [ ] 04-single-hop-6lowpan-icmp (@aabadie)
  • [x] Task #01 - ICMPv6 link-local echo with iotlab-m3
  • [x] Task #02 - ICMPv6 multicast echo with iotlab-m3/samr21-xpro
  • [x] Task #03 - ICMPv6 echo with large payload
  • [x] Task #04 - ICMPv6 echo with iotlab-m3/samr21-xpro 15 minutes
  • [ ] Task #05 (Experimental) - ICMPv6 multicast echo with samr21-xpro/remote
  • [ ] Task #06 (Experimental) - ICMPv6 link-local echo with samr21-xpro/remote
  • [x] Task #07 (Experimental) - ICMPv6 multicast echo with samr21-xpro/zero + xbee
  • [x] Task #08 (Experimental) - ICMPv6 echo with samr21-xpro/zero + xbee
  • [x] Task #09 - ICMPv6 stress test
• [x] 05-single-hop-route (@aabadie)
  • [x] Task #01 - ICMPv6 echo unicast addresess on native (default route)
  • [x] Task #02 - ICMPv6 echo unicast addresess on iotlab-m3 (default route)
  • [x] Task #03 - ICMPv6 echo unicast addresess on native (specific route)
  • [x] Task #04 - ICMPv6 echo unicast addresess on iotlab-m3 (static route)
• [x] 06-single-hop-udp
  • [x] Task #01 - UDP on iotlab-m3 (@leandrolanzieri)
  • [x] Task #02 - UDP on iotlab-m3 (UDP port compression)
  • [x] Task #03 - UDP on native (non-existent neighbor)
  • [x] Task #04 - UDP on iotlab-m3 (non-existent neighbor)
• [x] 07-multi-hop (@cgundogan)
  • [x] Task #01 - ICMPv6 echo on iotlab-m3 with three hops (static route)
  • [x] Task #02 - UDP on iotlab-m3 with three hops (static route)
  • [x] Task #03 - ICMPv6 echo on iotlab-m3 with three hops (RPL route)
  • [x] Task #04 - UDP on iotlab-m3 with three hops (RPL route)
• [x] 08-interop
  • [X] Task #01 - ICMPv6 echo between native and Linux (@cgundogan)
  • [X] Task #02 - ICMPv6 echo between iotlab-m3 and Linux with 6LowPAN (@cgundogan)
  • [x] Task #03 - ICMPv6 echo between iotlab-m3 and Contiki
  • [x] Task #04 - ICMPv6 echo between iotlab-m3 and Internet host through Linux with 6LowPAN
  • [x] Task #05 - ICMPv6 echo between iotlab-m3 and Internet host through RIOT border router
  • [x] Task #06 - UDP between iotlab-m3 and Internet host through RIOT border router
  • [x] Task #07 - UDP between iotlab-m3 and Internet host through RIOT border router (200b payload)
  • [x] Task #08 - UDP between GNRC and lwIP on iotlab-m3
  • [x] Task #09 - UDP between GNRC and emb6 on iotlab-m3
  • [x] Task #10 - UDP between lwIP and emb6 on iotlab-m3
• [x] 09-coap (@kb2ma)
  • [x] Task #01 - CORD Endpoint
  • [x] Task #02 - Confirmable retries
  • [x] Task #03 - Block1
  • [x] Task #04 - Block2
  • [x] Task #05 - Observe registration and notification
• [x] 10-icmpv6-error (@miri64)
  • [x] Task #01 - Destination unreachable - no route to destination
  • [x] Task #02 - Destination unreachable - Beyond scope of source address
  • [x] Task #03 - Destination unreachable - address unreachable (target node address)
  • [x] Task #04 - Destination unreachable - address unreachable (neighbor cache miss)
  • [x] Task #05 - Destination unreachable - port unreachable
  • [x] Task #06 - Destination unreachable - port unreachable (large payload)
  • [x] Task #07 - Packet too big
  • [x] Task #08 - Time exceeded - hop limit exceeded in transit
  • [x] Task #09 - Parameter problem - erroneous header field encountered
  • [x] Task #10 - IPv6-in-IPv6 encapsulation
• [ ] 99-compile-and-test-one-board
  • [ ] Task #01 - Run tests on different hardwares

[miri64]

I started running Task 1.1. Should be done during the weekend.

[kb2ma]

09-coap runs fine on native.

[miri64]

10-icmpv6-error succeeded, however I found some errors in its testing procedures (fixed in #93) and an unrelated bug (caused by following the incorrect testing procedures) in RIOT itself (fixed in https://github.com/RIOT-OS/RIOT/pull/10879).

[aabadie]

I ran the automatic python script for tasks 2.2 and tasks 2.3 and got the following tests failing:

• tests/gnrc_ipv6_ext: needs to be root and maybe another setup is required to able to reach the node from the script (ethos, or something)
• tests/gnrc_rpl_srh: same as the first one
• tests/pkg_fatfs_vfs: needs some MTD setup for these boards which is not provided by default.

All other tests succeeded.

[cgundogan]

I successfully ran all multi-hop tests (following the description in https://github.com/RIOT-OS/Release-Specs/pull/85, btw)

[miri64]

tests/gnrc_ipv6_ext: needs to be root and maybe another setup is required to able to reach the node from the script (ethos, or something)

You need scapy and ethos for it. Both require sudo (ethos because the start_network.sh script is a little bit screwed up) and scapy because it needs to be able to send Ethernet frames.

[leandrolanzieri]

Successfully ran the single-hop-udp tests

[cgundogan]

I updated the progress list above, since some of my modifications removed @leandrolanzieri 's checks. We really need to find another way of documenting our progress ..

[miri64]

I started running Task 1.1. Should be done during the weekend.

All but gnrc_sixlowpan_frag succeeded

        Building application: gnrc_sixlowpan_frag (122/252)
                : acd52832 ... success., airfy-beacon ... success., arduino-due ... success., arduino-duemilanove ... success., arduino-mega2560 ... success., arduino-mkr1000 ... success., arduino-mkrfox1200 ... success., arduino-mkrzero ... success., arduino-uno ... success., arduino-zero ... success., avsextrem ... success., b-l072z-lrwan1 ... success., b-l475e-iot01a ... success., blackpill ... success., bluepill ... success., calliope-mini ... success., cc2538dk ... success., cc2650-launchpad ... success., cc2650stk ... success., chronos ... failed!, ek-lm4f120xl ... success., esp32-mh-et-live-minikit ... success., esp32-olimex-evb ... success., esp32-wemos-lolin-d32-pro ... success., esp32-wroom-32 ... success., esp32-wrover-kit ... success., esp8266-esp-12x ... success., esp8266-olimex-mod ... success., esp8266-sparkfun-thing ... success., f4vi1 ... success., feather-m0 ... success., firefly ... success., fox ... success., frdm-k22f ... success., frdm-k64f ... success., frdm-kw41z ... success., hifive1 ... success., ikea-tradfri ... success., iotlab-a8-m3 ... success., iotlab-m3 ... success., jiminy-mega256rfr2 ... success., limifrog-v1 ... success., lobaro-lorabox ... success., maple-mini ... success., mbed_lpc1768 ... success., mega-xplained ... success., microbit ... success., mips-malta ... success., msb-430 ... failed!, msb-430h ... failed!, msba2 ... success., msbiot ... success., mulle ... success., native ... success., nrf51dk ... success., nrf51dongle ... success., nrf52840-mdk ... success., nrf52840dk ... success., nrf52dk ... success., nrf6310 ... success., nucleo-f030r8 ... success., nucleo-f031k6 ... success., nucleo-f042k6 ... success., nucleo-f070rb ... success., nucleo-f072rb ... success., nucleo-f091rc ... success., nucleo-f103rb ... success., nucleo-f207zg ... success., nucleo-f302r8 ... success., nucleo-f303k8 ... success., nucleo-f303re ... success., nucleo-f303ze ... success., nucleo-f334r8 ... success., nucleo-f401re ... success., nucleo-f410rb ... success., nucleo-f411re ... success., nucleo-f412zg ... success., nucleo-f413zh ... success., nucleo-f429zi ... success., nucleo-f446re ... success., nucleo-f446ze ... success., nucleo-f722ze ... success., nucleo-f746zg ... success., nucleo-f767zi ... success., nucleo-l031k6 ... success., nucleo-l053r8 ... success., nucleo-l073rz ... success., nucleo-l152re ... success., nucleo-l432kc ... success., nucleo-l433rc ... success., nucleo-l452re ... success., nucleo-l476rg ... success., nucleo-l496zg ... success., nz32-sc151 ... success., opencm904 ... success., openmote-b ... success., openmote-cc2538 ... success., pba-d-01-kw2x ... success., phynode-kw41z ... success., pic32-clicker ... success., pic32-wifire ... success., remote-pa ... success., remote-reva ... success., remote-revb ... success., ruuvitag ... success., samd21-xpro ... success., saml10-xpro ... success., saml11-xpro ... success., saml21-xpro ... success., samr21-xpro ... success., samr30-xpro ... success., seeeduino_arch-pro ... success., sensebox_samd21 ... success., slstk3401a ... success., slstk3402a ... success., sltb001a ... success., slwstk6000b ... success., slwstk6220a ... success., sodaq-autonomo ... success., sodaq-explorer ... success., sodaq-one ... success., sodaq-sara-aff ... success., spark-core ... success., stk3600 ... success., stk3700 ... success., stm32f0discovery ... success., stm32f3discovery ... success., stm32f429i-disc1 ... success., stm32f4discovery ... success., stm32f769i-disco ... success., stm32l476g-disco ... success., teensy31 ... success., telosb ... failed!, thingy52 ... success., ublox-c030-u201 ... success., udoo ... success., usb-kw41z ... success., waspmote-pro ... success., wsn430-v1_3b ... failed!, wsn430-v1_4 ... failed!, yunjia-nrf51822 ... success., z1 ... failed!
                make buildtest error!

Will investigate what's wrong.

[miri64]

I started running Task 1.1. Should be done during the weekend.

All but gnrc_sixlowpan_frag succeeded

        Building application: gnrc_sixlowpan_frag (122/252)
                : acd52832 ... success., airfy-beacon ... success., arduino-due ... success., arduino-duemilanove ... success., arduino-mega2560 ... success., arduino-mkr1000 ... success., arduino-mkrfox1200 ... success., arduino-mkrzero ... success., arduino-uno ... success., arduino-zero ... success., avsextrem ... success., b-l072z-lrwan1 ... success., b-l475e-iot01a ... success., blackpill ... success., bluepill ... success., calliope-mini ... success., cc2538dk ... success., cc2650-launchpad ... success., cc2650stk ... success., chronos ... failed!, ek-lm4f120xl ... success., esp32-mh-et-live-minikit ... success., esp32-olimex-evb ... success., esp32-wemos-lolin-d32-pro ... success., esp32-wroom-32 ... success., esp32-wrover-kit ... success., esp8266-esp-12x ... success., esp8266-olimex-mod ... success., esp8266-sparkfun-thing ... success., f4vi1 ... success., feather-m0 ... success., firefly ... success., fox ... success., frdm-k22f ... success., frdm-k64f ... success., frdm-kw41z ... success., hifive1 ... success., ikea-tradfri ... success., iotlab-a8-m3 ... success., iotlab-m3 ... success., jiminy-mega256rfr2 ... success., limifrog-v1 ... success., lobaro-lorabox ... success., maple-mini ... success., mbed_lpc1768 ... success., mega-xplained ... success., microbit ... success., mips-malta ... success., msb-430 ... failed!, msb-430h ... failed!, msba2 ... success., msbiot ... success., mulle ... success., native ... success., nrf51dk ... success., nrf51dongle ... success., nrf52840-mdk ... success., nrf52840dk ... success., nrf52dk ... success., nrf6310 ... success., nucleo-f030r8 ... success., nucleo-f031k6 ... success., nucleo-f042k6 ... success., nucleo-f070rb ... success., nucleo-f072rb ... success., nucleo-f091rc ... success., nucleo-f103rb ... success., nucleo-f207zg ... success., nucleo-f302r8 ... success., nucleo-f303k8 ... success., nucleo-f303re ... success., nucleo-f303ze ... success., nucleo-f334r8 ... success., nucleo-f401re ... success., nucleo-f410rb ... success., nucleo-f411re ... success., nucleo-f412zg ... success., nucleo-f413zh ... success., nucleo-f429zi ... success., nucleo-f446re ... success., nucleo-f446ze ... success., nucleo-f722ze ... success., nucleo-f746zg ... success., nucleo-f767zi ... success., nucleo-l031k6 ... success., nucleo-l053r8 ... success., nucleo-l073rz ... success., nucleo-l152re ... success., nucleo-l432kc ... success., nucleo-l433rc ... success., nucleo-l452re ... success., nucleo-l476rg ... success., nucleo-l496zg ... success., nz32-sc151 ... success., opencm904 ... success., openmote-b ... success., openmote-cc2538 ... success., pba-d-01-kw2x ... success., phynode-kw41z ... success., pic32-clicker ... success., pic32-wifire ... success., remote-pa ... success., remote-reva ... success., remote-revb ... success., ruuvitag ... success., samd21-xpro ... success., saml10-xpro ... success., saml11-xpro ... success., saml21-xpro ... success., samr21-xpro ... success., samr30-xpro ... success., seeeduino_arch-pro ... success., sensebox_samd21 ... success., slstk3401a ... success., slstk3402a ... success., sltb001a ... success., slwstk6000b ... success., slwstk6220a ... success., sodaq-autonomo ... success., sodaq-explorer ... success., sodaq-one ... success., sodaq-sara-aff ... success., spark-core ... success., stk3600 ... success., stk3700 ... success., stm32f0discovery ... success., stm32f3discovery ... success., stm32f429i-disc1 ... success., stm32f4discovery ... success., stm32f769i-disco ... success., stm32l476g-disco ... success., teensy31 ... success., telosb ... failed!, thingy52 ... success., ublox-c030-u201 ... success., udoo ... success., usb-kw41z ... success., waspmote-pro ... success., wsn430-v1_3b ... failed!, wsn430-v1_4 ... failed!, yunjia-nrf51822 ... success., z1 ... failed!
                make buildtest error!

Will investigate what's wrong.

Ok, the application gets too large for MSP430 platforms on my machine, but on Murdock it is small enough (see build of https://github.com/RIOT-OS/RIOT/pull/10886 e.g.), so not really a problem.

[miri64]

We really need to find another way of documenting our progress ..

Maybe a reusable project could help with that?

[aabadie]

Task 4.4 is failing, packet loss is above 10%

[cgundogan]

Task 4.4 is failing, packet loss is above 10%

@aabadie I am running this task using a local samr and iotlab. Will report back in ~15min.

[cgundogan]

2019-01-28 18:47:48,865 - INFO # --- fe80::1711:6b10:65fa:a932 ping statistics --- 2019-01-28 18:47:48,871 - INFO # 10000 packets transmitted, 9991 received, 1% packet loss, time 1313.06522340 s 2019-01-28 18:47:48,874 - INFO # rtt min/avg/max = 17.590/22.397/46.515 ms

@aabadie this is what I get with a local setup. Maybe you have some interferences on channel 26 at your site?

[aabadie]

Maybe you have some interferences on channel 26 at your site?

Maybe yes, I'll rerun the tests on another channel.

[miri64]

I re-did 10.07 with https://github.com/RIOT-OS/Release-Specs/pull/96 included. Still passes.

[aabadie]

I'll rerun the tests on another channel.

They are passing ~5% packet loss.

[aabadie]

Tasks 05 are passing. I also ran tasks 04 (except 04.05 and 04.06 because I don't have the hardware) with success.

[aabadie]

Also completed tasks 1.3 and 1.4 with success.

[jia200x]

tasks 05, 06 and 07 from 08.interop run OK

[jia200x]

Interop tests between GNRC, emb6 and lwip pass

[aabadie]

I ran 99 on nrf52dk, samr21-xpro and iotlab-m3 and got the same results as reported in https://github.com/RIOT-OS/Release-Specs/issues/92#issuecomment-457928353

[aabadie]

Is there someone who could test 4.5, 4.6, 8.3 and 8.4 ? After that, we are ready for the RC2

[miri64]

Nothing unexpected to report for 8.4.

[miri64]

I finally was able to do 8.3. I based it on @kYc0o comment here https://github.com/RIOT-OS/Release-Specs/issues/58#issuecomment-361926531. It was still not that straightforward. I will try to update the specs and test descriptions.

[miri64]

I finally was able to do 8.3. I based it on @kYc0o comment here #58 (comment). It was still not that straightforward. I will try to update the specs and test descriptions.

https://github.com/RIOT-OS/Release-Specs/pull/97

[aabadie]

Closing in favor of #98

[aabadie]

Thanks @miri64 for running the missing tasks!