ekiwi
commented
2 years ago Hi Nic,
First question: can the default simulator backend be set in config file (e.g., build.sbt)?
Not directly, since I did not want to open the can of worms that is configuration management. However, you could fairly easily built this functionality into your own test setup. In fact, for chiseltests, we are using a flag to select the default backend for formal tests, when we run our own unittests. See: https://github.com/ucb-bar/chiseltest/blob/master/src/test/scala/chiseltest/formal/FormalBackendOption.scala
Second question: is there any reason my thinking on this methodology is wrong?
Your methodology seems fine to me if you are thinking about running tests once or twice or only on one machine. However, one of the major test cases for chiseltest is continous integration. For that to work you need some way to remember the best simulator to use that will be available to everyone using your repository. Adding the explicit annotation is an easy way to do this. If you did not remember which simulator to use, then every CI run would have to figure out from scratch which one to use and that would make CI take longer.
Another thing to consider is that designs, simulators and testbenches change. Only because currently treadle performs best does not mean that this will be the case with a newer version of the components involved. So you would have to come up with a clever system to invalidate your "preferred simulator cache".
One more thing to consider is that while we aim for Treadle and Verilator to be interchangeable, that may not always be the case because of bugs, or because of tool specific features, like Verilog black boxes.
Third question: could another backend be created that simply passes off the test calls to Treadle or Verilator as described above?
That should generally be feasible.
All in all I think that this could be a good idea under certain situations and may be interesting to certain users. However, it might be too complicated / brittle to make it work well across a wider number of users all of which have different requirements.
I really don't care whether treadle or verilator is used as my simulation backend. The only thing I care about is accuracy and speed. Assuming they are both accurate, then the only concern is speed. The main trade-off is simply the start up cost of Verilator.
Currently I write tests. If tests take too long to complete (>15s) then I add the Verilator backend annotation. A cool feature would be a system for remembering how long tests take and choosing the appropriate backend to minimize time execution. A secondary feature would be to dynamically cancel a test if a tests time passes a threshold (say 15 seconds) then restart it with Verilator instead. The next time the tests are run it would know to choose Verilator. If a test every shrinks in time (say below 10 seconds), then the automatic backend selection feature would choose Treadle the next time.
First question: can the default simulator backend be set in config file (e.g., build.sbt)?
Second question: is there any reason my thinking on this methodology is wrong?
Third question: could another backend be created that simply passes off the test calls to Treadle or Verilator as described above?