How to use VUnit in a larger framework through a lower-level interface

[Paebbels]

The PoC-Library contains her own:

• source file dependency management
• simulator abstraction layer
• testbench naming scheme

so no high-level run.py is needed. Using the user API intended for run.py would double the work. I'm looking for low-level APIs to get access to VUnits features like:

• VHDL runner
  • running single test cases
  • running test cases in parallel
  • running test cases in sequence
  • restart failing test cases
• jUnit report files
  • these files shall be readable by GitLab (planned for 10.5 - 22.02.2018)

My initial question might be to broad, so please direct me to create a good issue also for other users as a reference.

Main questions:

• How to create the runner_cfg generic for VHDL test benches?
  • What information are encoded in that generic?
• How does a VUnit testbench communicate back to the Python scripts?

I found important code at lines 194-214 in test_suites.py.

[kraigher]

Regarding runner_cfg

The runner_cfg is a dictionary with the following fields:

• "enabled_test_cases"
  • A list of test case names to run or "__all__" meaning all test cases
• "output path"
  • The path to the output directory for the test. Each individual test gets a separate output directory.
• "active python runner"
  • Always True when running using external automation.
• "tb path"
  • The path to the folder containing the test bench. NOTE: Note the space in some of the dictionary keys. I know it is weird and we should probably change that.

On the branch there is also a "use_color" field to enable or disable ANSI color output. This is set to true in the terminal and false in the simulator GUI:s that do not handle ANSI color codes (which is all of them).

Regarding test results

The VHDL runner places a file named vunit_results in the test output path. For each started test it prints test_start:<Test name> and when test_runner_cleanup has been successfully run it writes test_suite_done. If the simulation ends without test_suite_done the latest started test is failed.

General thoughts

In a VUnit run.py file users can normally set compile options and simulation options. They can also configure the same test bench to run for multiple generic values. If you do not use the public API and also your own simulator abstraction this will not be available to you. VUnit also supports comment pragmas that can be placed in the test bench such as fail_on_warning run_all_in_same_sim. In the future we might add more pragmas for configuring runs with different generics and simulation options as well as compile options. Depending on you integrate this or the perspective of your users they might find some "normal" features of VUnit broken or missing in your environment.

It would be easier for you and more stable if you can re-use https://github.com/VUnit/vunit/blob/master/vunit/test_bench_list.py and https://github.com/VUnit/vunit/blob/master/vunit/test_bench.py. These classes parse the test bench files for tests and pragmas and creates the test suite classes where the runner_cfg creation and vunit_results parsing is done.

You just have to pass it your own simulator_if instance that has a simulate method that conforms with the API used by VUnit.

[Paebbels]

Thanks for the explanation. I'll look into the references files.

---

Why don't you implement the "comment pragmas" with VHDL tool directives? These directives are designed for such a purpose.

VHDL-2008 pre-defines the protect directives and VHDL-2018 adds conditional analysis. You could create your own VUnit pragmas.

`VUnit fail_on_warning     true
`VUnit run_all_in_same_sim true

entity tb is
end entity;

It would be nice to know, how tolerant simulators are for unknown tool directives.

[kraigher]

Regarding VHDL tool directives I do not want to create compatability issues just to use a new feature of the VHDL standard when there is an alternative that will work everywhere.

[kraigher]

@Paebbels In another forum you also mentioned that you wanted to run testcases using different simulators within the same test suite. I then answered that would be an easy addition to VUnit but so far no one has asked for it. As you can see the TestBenchList and TestBench classes I referred you to get a simulator_if instance so it is already an injected dependency. You only have to run the test case creation methods for every simulator instance to accumulate them in the test suite. The only missing feature would be to support an additional flag to add a prefix on the test name with the simulator name such as:

ghdl:lib.test_bench.test_case
rivierapro:lib.test_bench.test_case

Instead of

lib.test_bench.test_case

[Paebbels]

We are currently using these command names:

• Simulators
  • asim - Active-HDL
  • rpro - Riviera-PRO
  • msim - ModelSim
  • qsim - QuestaSim
  • vsim - the configured simulator, that is compatible to the vlib, vmap, vcom, vsim CLI
  • ghdl - GHDL
  • isim - ISE Simulator
  • xsim - Vivado Simulator
• Synthesis
  • quartus - Altera Quartus II / Intel Quartus Prime
  • xst - XST toolchain from ISE
  • coregen - CoreGenerattor from ISE
  • ise - resolved dependencies between pre-generated IP cores, patching and netlist synthesis
  • vivado - Vivado synthesis
  • lse - Diamond synthesis (Lattice Synthesis Engine)

[kraigher]

@Paebbels what is the status of this topic?

[Paebbels]

I'll have some free days next week and I'm currently preparing a big new release for PoC.
Then I'll work on that topic again, because I would like to present this at FPGA Kongress.

So it's still an ongoing topic.