[bazel] Output Verilator run command including paths to all binaries

[vogelpi]

As discussed in #14353 , it would be very useful if bazel would output the command for running the Verilator simulation of Earl Grey including paths to all binaries. I see the following two main advantages especially important when it comes to chip-level tests:

1. When working on software changes, users can easily extract the Verilator simulation binary and then rerun the test manually, i.e., bazel is only used to recompile the software binaries but not for rerunning. This is useful as bazel frequently decides to rebuild the Verilator simulation which can take a lot of time. See #14353
2. When manually launching it, one can use the kill command for enabling/disabling tracing while the Verilator simulation is running. Tracing slows down the simulation by several orders of magnitude. Thus it's better to only enable it after ROM. When launching the test through bazel, this doesn't seem to work and tracing has to be enabled right from the start which is very time consuming.

Currently bazel prints something like this to the console:

INFO: Found 1 test target...
Invoking test: sw/host/opentitantool/opentitantool --rcfile= --logging=info --interface=verilator --verilator-bin=hw/build.verilator_real/sim-verilator/Vchip_sim_tb --verilator-rom=sw/device/lib/testing/test_rom/test_rom_sim_verilator.39.scr.vmem --verilator-flash=sw/device/tests/aes_smoketest_prog_sim_verilator.test_key_0.signed.64.scr.vmem --verilator-otp=hw/ip/otp_ctrl/data/img_rma.24.vmem console --exit-success=PASS.*\n --exit-failure=((FAIL|FAULT).*\n)|(BFV:[0-9a-f]{8})

But what we would need is rather something like this:

bazel-bin/hw/build.verilator_real/sim-verilator/Vchip_sim_tb \
--meminit=rom,bazel-bin/sw/device/tests/aes_smoketest_sim_verilator.runfiles/lowrisc_opentitan/sw/device/lib/testing/test_rom/test_rom_sim_verilator.39.scr.vmem \
--meminit=otp,bazel-bin/sw/device/tests/aes_smoketest_sim_verilator.runfiles/lowrisc_opentitan/hw/ip/otp_ctrl/data/img_rma.24.vmem \
--meminit=flash,bazel-bin/sw/device/tests/aes_smoketest_sim_verilator.runfiles/lowrisc_opentitan/sw/device/tests/aes_smoketest_prog_sim_verilator.64.scr.vmem

@drewmacrae do you think this would be possible?

[a-will]

We use the term "Verilator" in rather confusing ways, don't we? Verilator is the toolchain that converts System Verilog to a C++ simulation model, and we add our own C++ code to that to create an actual simulator (which doesn't seem to have a good name, unless you want to go with "Vchip_sim_tb" :joy: ).

In any case, bazel can output the commands it calls via the --subcommands argument.

However, for building the model, it's actually invoking fusesoc (rather than Verilator directly), and for running tests, it calls util/opentitan_functest_runner.sh, which calls opentitantool, which finally spawns the simulator that drives the verilated model. It looks like opentitantool does output its invocation at info log level: https://cs.opensource.google/opentitan/opentitan/+/master:sw/host/opentitanlib/src/transport/verilator/subprocess.rs;l=53;drc=cba8148c0691d2940b955fd960d751e4ae27af18

Are you running the test with bazel test? If you'd like to see the stdout of tests in real time, try --test_output=streamed. The default is "summary," which only gives a status at the end. Here's an example of what you get with "streamed":

Invoking test: sw/host/opentitantool/opentitantool --rcfile= --logging=info --interface=verilator --verilator-bin=hw/build.verilator_real/sim-verilator/Vchip_sim_tb --verilator-rom=sw/device/lib/testing/test_rom/test_rom_sim_verilator.39.scr.vmem --verilator-flash=sw/device/tests/usbdev_test_prog_sim_verilator.test_key_0.signed.64.scr.vmem --verilator-otp=hw/ip/otp_ctrl/data/img_rma.24.vmem console --exit-success=PASS.*\n --exit-failure=((FAIL|FAULT).*\n)|(BFV:[0-9a-f]{8})
[2022-09-23T15:23:09Z INFO  opentitanlib::transport::verilator::subprocess] Spawning verilator: "hw/build.verilator_real/sim-verilator/Vchip_sim_tb" ["--meminit=rom,sw/device/lib/testing/test_rom/test_rom_sim_verilator.39.scr.vmem", "--meminit=flash,sw/device/tests/usbdev_test_prog_sim_verilator.test_key_0.signed.64.scr.vmem", "--meminit=otp,hw/ip/otp_ctrl/data/img_rma.24.vmem"]
[2022-09-23T15:23:09Z INFO  opentitanlib::transport::verilator::stdout] Simulation of OpenTitan Earl Grey
[2022-09-23T15:23:09Z INFO  opentitanlib::transport::verilator::stdout] =================================

Notice the "Spawning verilator" line. The format is not the nicest for copy-pasting into the shell, though...

[vogelpi]

Thanks for the quick response. Yes, I am running with bazel test and using --test_output=streamed.

I've overlooked the Spawning Verilator line. Beyond the not ideal format, the main problem with that line is that the specified paths are wrong. E.g. the path to Vchip_sim_tb needs bazel-bin at the beginning, the OTP image is at bazel-bin/sw/device/tests/aes_smoketest_sim_verilator.runfiles/lowrisc_opentitan/hw/ip/otp_ctrl/data/img_rma.24.vmem instead of hw/ip/otp_ctrl/data/img_rma.24.vmem. This is not really usable at the moment.

[a-will]

Those paths actually aren't wrong, per se. However, sandboxing does make it obscure where the working directory is: https://bazel.build/reference/test-encyclopedia#initial-conditions

We could ostensibly have opentitan_functest_runner.sh print out the current working directory, so you can see where all relative paths originate. Would that help?

[vogelpi]

I just gave #15151 a try: on the console I can now see

[2022-09-29T10:40:27Z INFO  opentitanlib::transport::verilator::subprocess] CWD: Ok("/home/pirmin/.cache/bazel/_bazel_pirmin/1ad0bf7145ecb30f7035281cc77ff581/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/aes_smoketest_sim_verilator.runfiles/lowrisc_opentitan")
[2022-09-29T10:40:27Z INFO  opentitanlib::transport::verilator::subprocess] Spawning verilator: "hw/build.verilator_real/sim-verilator/Vchip_sim_tb" "--meminit=rom,sw/device/lib/testing/test_rom/test_rom_sim_verilator.39.scr.vmem --meminit=flash,sw/device/tests/aes_smoketest_prog_sim_verilator.test_key_0.signed.64.scr.vmem --meminit=otp,hw/ip/otp_ctrl/data/img_rma.24.vmem"

doing a

cd /home/pirmin/.cache/bazel/_bazel_pirmin/1ad0bf7145ecb30f7035281cc77ff581/execroot/lowrisc_opentitan/bazel-out/k8-fastbuild/bin/sw/device/tests/aes_smoketest_sim_verilator.runfiles/lowrisc_opentitan

and then

hw/build.verilator_real/sim-verilator/Vchip_sim_tb --meminit=rom,sw/device/lib/testing/test_rom/test_rom_sim_verilator.39.scr.vmem --meminit=flash,sw/device/tests/aes_smoketest_prog_sim_verilator.test_key_0.signed.64.scr.vmem --meminit=otp,hw/ip/otp_ctrl/data/img_rma.24.vmem

it is now very easy to rerun a simulation 1) with another, backed up Vchip_sim_tb, 2) where tracing can be enabled/disabled upon request with the kill command. This makes working with Verilator again a lot easier which is fantastic. Many thanks @drewmacrae , this is really appreciated!