search

riscv-software-src / riscof

BSD 3-Clause "New" or "Revised" License
63 stars 40 forks source link

Failing test stops reports from being generated #73

Closed gsmecher closed 1 year ago

gsmecher commented 1 year ago

The default template (Templates/setup/model/riscof_model.py) creates a Makefile and invokes it. The net result is a command along the lines of

make -j16 -f Makefile.DUT-thing TARGET0 TARGET1 TARGET2 TARGET3 TARGET4 TARGET5 TARGET6 TARGET7 TARGET8 TARGET9 TARGET10 TARGET11 TARGET12 TARGET13 TARGET14 TARGET15 TARGET16 TARGET17 TARGET18 TARGET19 TARGET20 TARGET21 TARGET22 TARGET23 TARGET24 TARGET25 TARGET26 TARGET27 TARGET28 TARGET29 TARGET30 TARGET31 TARGET32 TARGET33 TARGET34 TARGET35 TARGET36 TARGET37 TARGET38 TARGET39 TARGET40 TARGET41 TARGET42 TARGET43 TARGET44 TARGET45 TARGET46 TARGET47 TARGET48 TARGET49 TARGET50 TARGET51 TARGET52 TARGET53 TARGET54 TARGET55 TARGET56 TARGET57 TARGET58 TARGET59 TARGET60 TARGET61 TARGET62 TARGET63 TARGET64 TARGET65 TARGET66 TARGET67 TARGET68 TARGET69 TARGET70 TARGET71 TARGET72 TARGET73 TARGET74 TARGET75 TARGET76 TARGET77 TARGET78 TARGET79 TARGET80

Perfect - we run tests in parallel. However, because make is not invoked with the "-k" option, a failing test prevents subsequent tests from being queued and executed. For these tests, no signature file is written and the following step fails as follows:

INFO | Running Tests on Reference Model.
INFO | Initiating signature checking.
ERROR | Signature file : /path/to/DUT-thing.signature does not exist

This brings down the runner and no reports are generated, which is not intended behaviour.

The template should invoke make with the "-k" option, which tells make to continue queueing and executing unrelated jobs after a failure. The makeUtil class (riscof/utils.py) does not provide a mechanism to pass this option along, but it can be added directly to the template as follows:

--- a/riscof/Templates/setup/model/riscof_model.py
+++ b/riscof/Templates/setup/model/riscof_model.py
@@ -117,7 +117,7 @@ class dutname(pluginTemplate):

       # set the make command that will be used. The num_jobs parameter was set in the __init__
       # function earlier
-      make.makeCommand = 'make -j' + self.num_jobs
+      make.makeCommand = 'make -k -j' + self.num_jobs

       # we will iterate over each entry in the testList. Each entry node will be refered to by the
       # variable testname.

I have tried this in my version of the template script and it seems like a definite improvement: when a test fails, make continues anyways and a report is generated.