Perfect - we run tests in parallel. However, because make is not invoked with the "-k" option, a failing test prevents subsequent tests from being queued and executed. For these tests, no signature file is written and the following step fails as follows:
INFO | Running Tests on Reference Model.
INFO | Initiating signature checking.
ERROR | Signature file : /path/to/DUT-thing.signature does not exist
This brings down the runner and no reports are generated, which is not intended behaviour.
The template should invoke make with the "-k" option, which tells make to continue queueing and executing unrelated jobs after a failure. The makeUtil class (riscof/utils.py) does not provide a mechanism to pass this option along, but it can be added directly to the template as follows:
--- a/riscof/Templates/setup/model/riscof_model.py
+++ b/riscof/Templates/setup/model/riscof_model.py
@@ -117,7 +117,7 @@ class dutname(pluginTemplate):
# set the make command that will be used. The num_jobs parameter was set in the __init__
# function earlier
- make.makeCommand = 'make -j' + self.num_jobs
+ make.makeCommand = 'make -k -j' + self.num_jobs
# we will iterate over each entry in the testList. Each entry node will be refered to by the
# variable testname.
I have tried this in my version of the template script and it seems like a definite improvement: when a test fails, make continues anyways and a report is generated.
The default template (Templates/setup/model/riscof_model.py) creates a Makefile and invokes it. The net result is a command along the lines of
Perfect - we run tests in parallel. However, because
make
is not invoked with the "-k" option, a failing test prevents subsequent tests from being queued and executed. For these tests, no signature file is written and the following step fails as follows:This brings down the runner and no reports are generated, which is not intended behaviour.
The template should invoke make with the "-k" option, which tells make to continue queueing and executing unrelated jobs after a failure. The makeUtil class (riscof/utils.py) does not provide a mechanism to pass this option along, but it can be added directly to the template as follows:
I have tried this in my version of the template script and it seems like a definite improvement: when a test fails, make continues anyways and a report is generated.