search

Linaro / lite-lava-docker-compose

LITE Team LAVA docker dispatcher
MIT License
5 stars 10 forks source link

Results of some zephyr-upstream tests don't appear in SQUAD reports #79

Open pfalcon opened 4 years ago

pfalcon commented 4 years ago

Steps to reproduce:

  1. Visit https://qa-reports.linaro.org/lite/zephyr-upstream/build/636d6cd9-5277/?failures_only=false#!#test-results
  2. Search on the page for "loopback-drivers-spi".
  3. Behold that it failed for 5 of 6 configurations. But cell for 6th configuration, frdm_kw41z-zephyr, is just empty, it neither failed, nor succeeded.
  4. This is especially strange, given that frdm_kw41z-gnuarmemb was run (and failed). Certainly, if we build that test with gnuarmemb toolchain, then we also built it with Zephyr SDK? Built and submitted.
  5. Let's try to check that, xrefing to the corresponding Jenkins job: https://ci.linaro.org/job/zephyr-upstream/PLATFORM=frdm_kw41z,ZEPHYR_TOOLCHAIN_VARIANT=zephyr,label=docker-xenial-amd64-13/5277/console
  6. Grep for "spi_loopback" (names are differing between "test case names" as shown in SQUAD, and "test file names", that's not helpful).
  7. Seen: 
    00:58:25.880 Uploading file /home/buildslave/workspace/zephyr-upstream/out/frdm_kw41z/tests/drivers/spi/spi_loopback/drivers.spi/zephyr/zephyr.bin
01:02:18.032 tests/drivers/spi/spi_loopback/drivers.spi/zephyr/zephyr.bin

    I.e., the test file was uploaded to the publishing server, and it was selected for submitting a job to LAVA.

Sadly, there's no easy away to cross-reference that to the actual submitted job. Let me pause here and improve logging in this regard.

pfalcon commented 4 years ago

https://review.linaro.org/c/ci/job/configs/+/35649

pfalcon commented 4 years ago

@vanti : FYI about this ticket.

vanti commented 4 years ago

Looks like this corresponds to an incomplete job: https://validation.linaro.org/scheduler/job/2023159

pfalcon commented 4 years ago

Looks like this corresponds to an incomplete job: https://validation.linaro.org/scheduler/job/2023159

Thanks @vanti for tracing this down! @mwasilew, can you please confirm that's the intended behavior (and vice-versa, if we see a "hole" in the 2D chart, the likely explanation is that it corresponds to "Incomplete" LAVA job)?

mwasilew commented 4 years ago

@pfalcon yes and no. Incomplete lava jobs are ignored. But if you don't submit any job, there will also be a 'hole'.

pfalcon commented 4 years ago

@pfalcon yes and no. Incomplete lava jobs are ignored. But if you don't submit any job, there will also be a 'hole'.

@mwasilew, which leads us to "tentative" feature request that it would be nice to draw these "errored" (per terminology you propose in https://github.com/Linaro/squad/issues/753) in e.g. grey. Here it's of course a special case of it, as test run == 1 test suite, in a general case it wouldn't be that easy (well, it's not easy here either, because for "incomplete" run, there's no information of what testsuite is there).