RIOT-OS / Release-Specs

Specification for RIOT releases and corresponding test configurations
4 stars 21 forks source link

Release 2022.10 - RC1 #261

Closed miri64 closed 2 years ago

miri64 commented 2 years ago

This issue lists the status of all tests for the Release Candidate 1 of the 2022.10 release.

Specs tested:

riot-ci commented 2 years ago

Test Report

Task Outcome
01. Task 04 - Unittests on iotlab-m3 PASSED
03. Task 03 - ICMPv6 link-local echo on native (1 hour) PASSED
03. Task 04 - ICMPv6 stress test on native (1 hour) PASSED
03. Task 05 - ICMPv6 stress test on native (neighbor cache stress) PASSED
03. Task 06 - ICMPv6 link-local echo on native (IPv6 fragmentation) PASSED
04. Task 01 - ICMPv6 link-local echo with iotlab-m3 PASSED
04. Task 02 - ICMPv6 multicast echo with iotlab-m3/samr21-xpro PASSED
04. Task 03 - ICMPv6 echo with large payload PASSED
04. Task 04 - ICMPv6 echo with iotlab-m3/samr21-xpro 15 minutes PASSED
🟡 04. Task 05 (Experimental) - ICMPv6 multicast echo with samr21-xpro/cc2538 [lille] SKIPPED
🟡 04. Task 06 (Experimental)- ICMPv6 link-local echo with samr21-xpro/cc2538 [lille] SKIPPED
🟡 04. Task 07 (Experimental) - ICMPv6 multicast echo with samr21-xpro/zero + xbee SKIPPED
🟡 04. Task 08 (Experimental) - ICMPv6 echo with samr21-xpro/zero + xbee SKIPPED
04. Task 09 - ICMPv6 stress test on iotlab-m3 PASSED
04. Task 10 (Experimental) - ICMPv6 echo with large payload (IPv6 fragmentation) PASSED
04. Task 11 (Experimental) - ICMPv6 stress test on nrf802154 PASSED
04. Task 12 (Experimental) - ICMPv6 multicast echo with iotlab-m3/nrf802154 PASSED
04. Task 13 (Experimental) - ICMPv6 link-local echo with iotlab-m3/nrf802154 PASSED
05. Task 01 - ICMPv6 echo unicast addresess on native (default route) PASSED
05. Task 02 - ICMPv6 echo unicast addresess on iotlab-m3 (default route) PASSED
05. Task 03 - ICMPv6 echo unicast addresess on native (specific route) PASSED
05. Task 04 - ICMPv6 echo unicast addresess on iotlab-m3 (static route) PASSED
06. Task 01 - UDP on iotlab-m3 PASSED
06. Task 02 - UDP on iotlab-m3 (UDP port compression) PASSED
06. Task 03 - UDP on native (non-existent neighbor) PASSED
06. Task 04 - UDP on iotlab-m3 (non-existent neighbor) PASSED
06. Task 05 - Empty UDP on native PASSED
06. Task 06 - Empty UDP on iotlab-m3 PASSED
07. Task 01 - ICMPv6 echo on iotlab-m3 with three hops (static route) PASSED
07. Task 02 - UDP on iotlab-m3 with three hops (static route) PASSED
07. Task 03 - ICMPv6 echo on iotlab-m3 with three hops (RPL route) PASSED
07. Task 04 - UDP on iotlab-m3 with three hops (RPL route) PASSED
🟡 07. Task 05 (Experimental) - UDP with large payload on iotlab-m3 with three hops (RPL route) SKIPPED
08. Task 01 - ICMPv6 echo between native and Linux PASSED
08. Task 08 - UDP between GNRC and lwIP on iotlab-m3 PASSED
09. Task 01 - CORD Endpoint PASSED
09. Task 02 - Confirmable retries [False-0] PASSED
09. Task 02 - Confirmable retries [True-1] PASSED
09. Task 03 - Block1 PASSED
09. Task 04 - Block2 PASSED
09. Task 05 - Observe registration and notification PASSED
10. Task 01 - Destination unreachable - no route to destination PASSED
10. Task 02 - Destination unreachable - Beyond scope of source address PASSED
10. Task 03 - Destination unreachable - address unreachable (target node address) PASSED
10. Task 04 - Destination unreachable - address unreachable (neighbor cache miss) PASSED
10. Task 05 - Destination unreachable - port unreachable PASSED
10. Task 06 - Destination unreachable - port unreachable (large payload) PASSED
10. Task 07 - Packet too big PASSED
10. Task 08 - Time exceeded - hop limit exceeded in transit PASSED
10. Task 09 - Parameter problem - erroneous header field encountered PASSED
10. Task 10 - IPv6-in-IPv6 encapsulation PASSED
11. Task 01 - LoRaWAN example PASSED
11. Task 02 - OTAA join procedure PASSED
11. Task 03 - ABP join procedure PASSED
11. Task 04 - LoRaWAN device parameters persistence PASSED
11. Task 05 - GNRC LoRaWAN (Over The Air Activation) [otaa] PASSED
🟡 11. Task 06 - GNRC LoRaWAN (ABP) [abp] SKIPPED
maribu commented 2 years ago

The LoRaWAN failure was expected, as https://github.com/RIOT-OS/RIOT/pull/18745 is not yet backported (and not even merged yet).

miri64 commented 2 years ago

The LoRaWAN failure was expected, as RIOT-OS/RIOT#18745 is not yet backported (and not even merged yet).

Nope, that fix does not have anything for that (according to @jia200x it is only about the dedicated tests/driver_* tests for those devices). The lorawan examples should still work, but there was a flashing error:

11-lorawan.test_spec11

Test case:test_task01[nodes0]
Outcome:Failed
Duration:43.141 sec
Failedsubprocess.CalledProcessError: Command '['make', '--no-print-directory', '-C', '/home/runner/work/RIOT/RIOT/RIOT/examples/lorawan', 'flash']' returned non-zero exit status 2.
riot_ctrl = <function riot_ctrl.<locals>.ctrl at 0x7f320083c040>
ttn_client = <testutils.ttn.TTNClient object at 0x7f320059e760>

    @pytest.mark.iotlab_creds
    # nodes passed to riot_ctrl fixture
    @pytest.mark.parametrize(
        "nodes", [pytest.param(["b-l072z-lrwan1"])], indirect=["nodes"]
    )
    def test_task01(riot_ctrl, ttn_client):
>       node = riot_ctrl(0, LORAWAN_APP, ShellInteraction)

11-lorawan/test_spec11.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
conftest.py:369: in ctrl
    node.make_run(
.tox/test/lib/python3.8/site-packages/riotctrl/ctrl.py:193: in make_run
    return subprocess.run(command, env=self.env, *runargs, **runkwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

input = None, capture_output = False, timeout = None, check = True
popenargs = (['make', '--no-print-directory', '-C', '/home/runner/work/RIOT/RIOT/RIOT/examples/lorawan', 'flash'],)
kwargs = {'env': {'APPEUI': '0000000000000000', 'APPKEY': '782D212CE97434EAB523EA7F98A70F52', 'APPSKEY': '2F4E8084C26812D4E95DE48305B65BC9', 'BOARD': 'b-l072z-lrwan1', ...}, 'stderr': None, 'stdout': None}
process = <subprocess.Popen object at 0x7f32008849a0>, stdout = None
stderr = None, retcode = 2

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['make', '--no-print-directory', '-C', '/home/runner/work/RIOT/RIOT/RIOT/examples/lorawan', 'flash']' returned non-zero exit status 2.

/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/subprocess.py:516: CalledProcessError
Stdout
--------------------------------- Captured Log ---------------------------------

--------------------------------- Captured Out ---------------------------------
Launching build container using image "docker.io/riot/riotbuild:latest".
docker run --rm --tty --user $(id -u) -v '/usr/share/zoneinfo/Etc/UTC:/etc/localtime:ro' -v '/home/runner/work/RIOT/RIOT/RIOT:/data/riotbuild/riotbase:delegated' -v '/home/runner/.cargo/registry:/data/riotbuild/.cargo/registry:delegated' -v '/home/runner/.cargo/git:/data/riotbuild/.cargo/git:delegated' -e 'RIOTBASE=/data/riotbuild/riotbase' -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' -e 'BUILD_DIR=/data/riotbuild/riotbase/build' -e 'RIOTPROJECT=/data/riotbuild/riotbase' -e 'RIOTCPU=/data/riotbuild/riotbase/cpu' -e 'RIOTBOARD=/data/riotbuild/riotbase/boards' -e 'RIOTMAKE=/data/riotbuild/riotbase/makefiles'      -e 'QUIETER=1' -e 'DEVEUI=70B3D57ED00463E7' -e 'APPEUI=0000000000000000' -e 'APPKEY=782D212CE97434EAB523EA7F98A70F52' -e 'BOARD=b-l072z-lrwan1' -e 'IOTLAB_NODE=st-lrwan1-11.saclay.iot-lab.info' -e 'DISABLE_MODULE=' -e 'DEFAULT_MODULE=' -e 'FEATURES_REQUIRED=' -e 'FEATURES_BLACKLIST=' -e 'FEATURES_OPTIONAL=periph_rtc' -e 'USEMODULE=auto_init_loramac fmt sx1276' -e 'USEPKG=semtech-loramac'  -w '/data/riotbuild/riotbase/examples/lorawan/' 'docker.io/riot/riotbuild:latest' make    -j  
#x1B[1;32mBuilding application "lorawan" for "b-l072z-lrwan1" with MCU "stm32".#x1B[0m

   text    data     bss     dec     hex filename
  56184     328    6340   62852    f584 /data/riotbuild/riotbase/examples/lorawan/bin/b-l072z-lrwan1/lorawan.elf
iotlab-node --jmespath='keys(@)[0]' --format='lambda ret: exit(int(ret))' --id 340908 --list saclay,st-lrwan1,11 --flash /home/runner/work/RIOT/RIOT/RIOT/examples/lorawan/bin/b-l072z-lrwan1/lorawan.bin

Stderr
--------------------------------- Captured Err ---------------------------------
make: *** [/home/runner/work/RIOT/RIOT/RIOT/Makefile.include:838: flash] Error 1

miri64 commented 2 years ago

For many of the problems with https://github.com/RIOT-OS/RIOT/actions/runs/3250328677 (test-on-iotlab) (which help cover 1.1, 2.1-3 and 99) https://github.com/RIOT-OS/RIOT/pull/18722 will help.

miri64 commented 2 years ago

heavy_multiplication_x 11. Task 02 - OTAA join procedure FAILED heavy_multiplication_x 11. Task 03 - ABP join procedure FAILED

Failed now, but succeeded in the last run.

leandrolanzieri commented 2 years ago

I re-triggered spec 11 task02 and task03 and it succeeded. See https://github.com/RIOT-OS/RIOT/actions/runs/3311670065/jobs/5467395615

leandrolanzieri commented 2 years ago

Closing in favor of #264