zephyr.git/tests/net/route/testcase.yaml#test :evalution failed

[nashif]

Reported by rahul kumar:

This testcase failed at evalution at particular on arduino board and Esp32.

the testcase failure report is pasted below :

Detailed description
====================
Automated test run failed.

Output (the target might have produced none; GDB will be needed at this point):

build passed: 'mkdir -p /home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/.run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101; rm -f /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/.run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101/.config' @/home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/.git//app_zephyr.py:134
build passed: 'make -j -C /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/net/route/ EXTRA_CFLAGS="-Werror -Wno-error=deprecated-declarations" KCPPFLAGS=-DTC_RUNID=-170831-1228-90:1czj BOARD=arduino_101 ARCH=x86  O=/home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/cf.run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101 initconfig' @/home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/.git/fl/app_zephyr.py:176
build output: Using /home/jenkins/workspace/zephyr-master-f-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/boards/x86/arduino_101/arduino_101_defconfig as base
build output: Merging /home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/include/test.config
build output: Merging prj.conf
build output: Merging /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101/arc_target.conf
build output: Merging /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101/boot_config.conf
build output: #
build output: # configuration written to .config
build output: #
build passed: 'make -j -C /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/net/route EXTRA_CFLAGS="-Werror -Wno-error=deprecated-declarations" KCPPFLAGS=-DTC_RUNID=-170831-1228-90:1czj BOARD=arduino_101 ARCH=x86  O=/home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101' @/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git//app_zephyr.py:206
build passed: 'mkdir -p /home/jenkins/workspace/zephyr-master-f-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/.run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss; rm -f /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss/.config' @/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git/app_zephyr.py:134
build passed: 'make -j -C /home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/booting/stub/ EXTRA_CFLAGS="-Werror -Wno-error=deprecated-declarations" KCPPFLAGS=-DTC_RUNID=-170831-1228-90:1czj BOARD=arduino_101_sss ARCH=arc  O=/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss initconfig' @/home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/it/app_zephyr.py:176
build output: Using /home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/boards/arc/arduino_101_sss/arduino_101_sss_defconfig as base
build output: Merging /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/include/test.config
build output: Merging prj.conf
build output: Merging /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/un170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss/arc_target.conf
build output: Merging /home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/un-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss/boot_stub_config.conf
build output: #
build output: # configuration written to .config
build output: #
build passed: 'make -j -C /home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/booting/stub EXTRA_CFLAGS="-Werror -Wno-error=deprecated-declarations" KCPPFLAGS=-DTC_RUNID=170831-1228-90:1czj BOARD=arduino_101_sss ARCH=arc  O=/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss' @/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/it/:206
         @local             build passed  type:arduino101
build output: # configuration written to .config
build output: #
build passed: 'make -j -C /home/jenkins/workspace/zephyr-master-0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/zephyr.git/tests/booting/stub EXTRA_CFLAGS="-Werror -Wno-error=deprecated-declarations" KCPPFLAGS=-DTC_RUNID=170831-1228-90:1czj BOARD=arduino_101_sss ARCH=arc  O=/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss' @/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/it/:206
         @local             build passed  type:arduino101
deployed (try 1/4) set([(u'kernel-x86', u'/home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/.run170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101/zephyr.bin'), (u'kernel-arc', u'/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss/zephyr.bin')])
deploy (1 tries) passed
deploy passed  type:arduino101
Reset
eval failed: expected console output 'RunID: 170831-1228-90:1czj' from console 'a101-16:default' NOT FOUND after 60.2 s
eval failed trace: Traceback (most recent call last):
eval failed trace:   File "/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git/tc.py", line 1478, in _decorated_fn
eval failed trace:     r = fn(*args, **kwargs)
eval failed trace:   File "/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git//tc.py", line 2971, in _method_run
eval failed trace:     return self.__method_trampoline_call(fname, fn, _type, targets)
eval failed trace:   File "/home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git//tc.py", line 2927, in __method_trampoline_call
eval failed trace:     r = getattr(self, fname)(*targets)
eval failed trace:   File "/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git//tc_zephyr_sanity.py", line 606, in eval
eval failed trace:     console = target.rt.get("console", None))
eval failed trace:   File "/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/it/tc.py", line 995, in expect
eval failed trace:     self.testcase.expecter.run()
eval failed trace:   File "/home/jenkins/workspace/zephyr-master--v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git//expecter.py", line 255, in run
eval failed trace:     r = f(*args)
eval failed trace:   File "/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/it//expecter.py", line 407, in console_rx_eval
eval failed trace:     { 'target': target, "console output": of })
eval failed trace: failed_e: (u"expected console output 'RunID: 170831-1228-90:1czj' from console 'a101-16:default' NOT FOUND after 60.2 s", {'console output': <open file u'/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/un-170831-1228-90-sG3S_J/v5cv/eval-buffers-00-0/console-jfsotc03a101-16:v5cv-default.log', mode 'a+' at 0x2b4f11465e40>, 'target': <.tc.target_c object at 0x2b4f114ea110>})
evaluation failed  type:arduino101
clean passed: 'rm -rf /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/runi-170831-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101' @/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git/pp_zephyr.py:256
rc  clean passed: 'rm -rf /home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/tmp/run31-1228-90-sG3S_J/v5cv/outdir-v5cv-1czj-arduino_101_sss' @/home/jenkins/workspace/zephyr-master-v0.11-branch/LABEL/verify/SHARD/2-3/ZEPHYR_GCC_VARIANT/zephyr/git/app_zephyr.py:256
clean passed  type:arduino101
COMPLETION failed  type:arduino101

(Imported from Jira ZEP-2590)

[nashif]

by Andy Ross:

Can you try this again? There's nothing here that indicates the actual test even ran. General consensus is that all failures of the form "expected console output 'RunID: ..." are due to hardware failures in the TCF backend and not things that should be in Jira.

[nashif]

by Inaky Perez-Gonzalez:

ESP32s are known to not produce any output randomly with any test case that is reported to pass in previous runs (or with the same when running it, let's say, five times in a row after resetting it), but we still do not know. It happens at a rate way bigger than any other target, so that is a known issue with them that is still being triaged. Let's set aside ESP32 for now.

Removing that, there is only one failure of this on a101 in the DB, so this could be, like Andy Ross said, a glitch on the serial capture in from TCF or the target being quirky that day (that happens too). If the failures are more consistent on only said target, we'll take a look at it as it might be it is funky.

If the failure is not consistent over runs, before reporting we need to be sure