search

RIOT-OS / RobotFW-tests

Includes tests for RIOT based on the Robot Framework
GNU Lesser General Public License v2.1
4 stars 13 forks source link

add jitter plot script #99

Closed pokgak closed 3 years ago

pokgak commented 3 years ago

This PR adds the plotting script for the jitter plots. Attached are the example full HTML result: jitter.zip

MrKevinWeiss commented 3 years ago

https://hil.riot-os.org/results/robotFW_tests/2/overview

MrKevinWeiss commented 3 years ago

Also seem to be OK

pokgak commented 3 years ago

hmm some of the boards not showing the graphs even though the tests not showing as failing on the test statistics. Example: https://hil.riot-os.org/results/robotFW_tests/2/nucleo-l432kc/tests_ztimer_benchmarks/report

MrKevinWeiss commented 3 years ago

https://ci.riot-os.org/hil/blue/organizations/jenkins/robotFW_tests/detail/robotFW_tests/2/pipeline

Ya, I guess we have to look into the logs of the run.

MrKevinWeiss commented 3 years ago

They exist in the artifacts...

MrKevinWeiss commented 3 years ago

It seems that something in the xlst script did not catch it for example arduino due is good and mega is not

ping @ozfox

``` Running stylesheet and saving result took 81 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_gpio/output.xml took 10 ms Running stylesheet and saving result took 130 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_gpio/output.xml took 10 ms Running stylesheet and saving result took 2313 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_spi/output.xml took 14 ms Running stylesheet and saving result took 107 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_spi/output.xml took 14 ms Running stylesheet and saving result took 3117 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_timer/output.xml took 5 ms Running stylesheet and saving result took 114 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_timer/output.xml took 5 ms Running stylesheet and saving result took 1256 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_uart/output.xml took 4 ms Running stylesheet and saving result took 106 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_periph_uart/output.xml took 4 ms Running stylesheet and saving result took 1183 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_xtimer_benchmarks/output.xml took 60 ms warning: failed to load external entity "/var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_xtimer_benchmarks/includes/overhead.html" Running stylesheet and saving result took 114 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_xtimer_benchmarks/output.xml took 61 ms Running stylesheet and saving result took 13887 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_xtimer_cli/output.xml took 0 ms Running stylesheet and saving result took 117 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_xtimer_cli/output.xml took 0 ms Running stylesheet and saving result took 263 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_ztimer_benchmarks/output.xml took 60 ms warning: failed to load external entity "/var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_ztimer_benchmarks/includes/overhead.html" Running stylesheet and saving result took 113 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-due/tests_ztimer_benchmarks/output.xml took 61 ms Running stylesheet and saving result took 13842 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_gpio/output.xml took 9 ms Running stylesheet and saving result took 123 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_gpio/output.xml took 9 ms Running stylesheet and saving result took 2185 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_i2c/output.xml took 5 ms Running stylesheet and saving result took 106 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_i2c/output.xml took 5 ms Running stylesheet and saving result took 1327 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_spi/output.xml took 14 ms Running stylesheet and saving result took 109 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_spi/output.xml took 14 ms Running stylesheet and saving result took 3149 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_timer/output.xml took 5 ms Running stylesheet and saving result took 108 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_timer/output.xml took 5 ms Running stylesheet and saving result took 1268 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_uart/output.xml took 4 ms Running stylesheet and saving result took 107 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_periph_uart/output.xml took 4 ms Running stylesheet and saving result took 1200 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_xtimer_benchmarks/output.xml took 61 ms Running stylesheet and saving result took 114 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_xtimer_benchmarks/output.xml took 60 ms Running stylesheet and saving result took 13888 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_xtimer_cli/output.xml took 0 ms Running stylesheet and saving result took 117 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_xtimer_cli/output.xml took 0 ms Running stylesheet and saving result took 249 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_ztimer_benchmarks/output.xml took 53 ms Running stylesheet and saving result took 111 ms Parsing stylesheet /var/jenkins_home/workspace/robotFW_tests/RobotFW-frontend/xsl/main.xsl took 0 ms Parsing document /var/jenkins_home/jobs/robotFW_tests/builds/2/archive/build/robot/arduino-mega2560/tests_ztimer_benchmarks/output.xml took 52 ms Running stylesheet and saving result took 12097 ms ```
ozfox commented 3 years ago

I was not able to reproduce it on my local setup and it seems that it magically started working: https://hil.riot-os.org/results/robotFW_tests/2/arduino-due/tests_xtimer_benchmarks/report

MrKevinWeiss commented 3 years ago

Only some boards are not showing the plot. https://hil.riot-os.org/results/robotFW_tests/13/arduino-mega2560/tests_xtimer_benchmarks/report

MrKevinWeiss commented 3 years ago

https://hil.riot-os.org/results/robotFW_tests/13/esp8266-esp-12x/tests_xtimer_benchmarks/report

ozfox commented 3 years ago

Yes, and that's completely correct according to the pages.xml config file. :facepalm: Don't ask me why this is still the case. I thought the plots will only be generated for specific boards when I developed it which was obviously wrong.

I'll fix it tomorrow and remove this config completely..

ozfox commented 3 years ago

Should be fixed with https://github.com/RIOT-OS/RobotFW-frontend/commit/8dea4e5bd9277f03f26e66d89e7a1d8d9d4d4c2c

MrKevinWeiss commented 3 years ago

https://hil.riot-os.org/results/robotFW_tests/21/overview

Everything is looking good. If rebased the z1 board would also be in. NO RESULT is a product of improper wiring as the periph_gpio tests fails the DUT_IC pin check.

I am happy, any last words @ozfox or @pokgak ?

pokgak commented 3 years ago

Just noticed that all of the plots for the board will not be shown if there is one failing. Might be something for future improvements. For now, I'm fine with it as it :)