search

maclobdell / mbed-test-runner

Simple Mbed OS test runner for running automated tests on multiple boards and toolchains
0 stars 2 forks source link

Problem at running generate_scorecards #6

Closed MarceloSalazar closed 6 years ago

MarceloSalazar commented 6 years ago

Few things:

1) It complains about test_results not being defined. I think this line requires extra tab. https://github.com/maclobdell/simple_mbed_test_runner/blob/create_scorecards/generate_scorecards.py#L96

2) I've specified a folder to analyze files using option "-f .\test_output\master" However, it looks only one folder is analyzed.

3) It's hard to get the overall picture of what happened after the test. Could we also generate a master json object that can later be exported using json2html module?

@maclobdell

maclobdell commented 6 years ago

for 1 & 2, I've made some fixes and need to verify. For 3, lets talk more.

MarceloSalazar commented 6 years ago

I've updated to master , commit 906e5a2fb0df90464f8e3a483bbdd07e61254748

I see a few things related to my previous comment

  1. Running the script inside mbed-os, but outside of test_output folder
C:\mbed\mbed-os-multi>..\simple_mbed_test_runner\generate_scorecards.py
mbed os ver: 2018-July-2
target: DISCO_F413ZH
Traceback (most recent call last):
  File "C:\mbed\simple_mbed_test_runner\generate_scorecards.py", line 106, in <module>
    main()
  File "C:\mbed\simple_mbed_test_runner\generate_scorecards.py", line 50, in main
    generate_scorecard("./", target, mbed_ver)
  File "C:\mbed\simple_mbed_test_runner\generate_scorecards.py", line 96, in generate_scorecard
    scorecard_data[toolchain] = test_results
UnboundLocalError: local variable 'test_results' referenced before assignment

It would be great if the script could search for tests results in json format

  1. Running the script and specifying a folder

C:\mbed\mbed-os-multi>..\simple_mbed_test_runner\generate_scorecards.py -f test_output\2018-July-7 mbed os ver: DISCO_F413ZH

Note, there are more folders with multiple json files.
Why did it stop on DISCO_F413ZH ?

3. Not an issue - but an enhancement request - to discuss separately ;-)

4. New (minor) issue - Running the script outside of an mbed-os repo:

C:\mbed\simple_mbed_test_runner>python generate_scorecards.py -h Traceback (most recent call last): File "generate_scorecards.py", line 16, in from tools.targets import TARGETS ImportError: No module named tools.targets



Note: not a valid case, as today we need to use the tools.
maclobdell commented 6 years ago

this ticket is really messy. Too many issues mixed together. I think a lot of it has to do with running the script from the right location, etc. Closing this ticket now. lets re-test the current state and add tickets for any new issues.