Building VUnit's verification components and OSVVMLibraries at the same time

[umarcor]

This is an spin-off coming from #754. That issue is about using OSVVMLibraries in VUnit, after compiling them externally (using OSVVM's .pro files, GHDL's vendor scripts or any other solution). It also covers builting OSVVMLibraries in a run.py script, while disabling (not building) VUnit's verification components. This one is a complementary issue to discuss how to actually build VUnit's verification components and OSVVMLibraries at the same time.

https://github.com/VUnit/vunit/issues/754#issuecomment-950287485 @umarcor What happens if we have the latest OSVVM as a submodule and that is included with add_osvvm or add_verification_components and then just the rest are compiled manually. Wouldn't that work?

@LarsAsplund, that's an interesting question. Yes, it needs to work, meaning it is technically possible and not complex to achieve for someone experienced with both VUnit and OSVVM. However, it is not easy to understand for an arbitrary newcomer.

• VUnit master is using OSVVM 2021.09, because I bumped it some days ago: https://github.com/VUnit/vunit/commit/d1956b0db3c94e99bb13d005dea8b32197581465.
• In OSVB, I am installing OSVVM using three different approaches:
  • GHDL's vendor scripts (bash).
  • OSVVM's .pro files (tclsh).
  • A VUnit run.py file (python).

OSVVM's .pro files and VUnit's run.py file can handle OSVVM 2021.09, but GHDL's vendor scripts were not updated yet, so they require OSVVM 2021.06 (see ghdl/ghdl#1900). Therefore, the default CI in OSVB is using OSVVM 2021.06 at the moment: https://github.com/umarcor/osvb/tree/main/mods. In that context, mixing VUnit's OSVVM core (2021.09) with the submoduled OSVVMLibraries (2021.06) is likely to crash. Hence, I had to do several modifications:

• Ensure that the checked out version of OSVVMLibraries is matching the version of OSVVM submoduled in VUnit. There is no programmatic procedure to achieve it. The user needs to read VUnit's changelog to find it out. It is possible to know the commit sha of the OSVVM core that VUnit is using, but that belongs to a submodule of OSVVMLibraries. I'm not sure you can find the commit in the "parent" that matches a bump of the "child". Therefore, it's mainly a manual step.
• Do not use GHDL's vendor scripts.
• Update the run.py script to add VU.add_verification_components() and not to add lib OSVVM manually.

This is the set of changes compared to the previous test: https://github.com/umarcor/osvb/compare/bump-osvvm?expand=1

The CI result is the same using this approach or the previous one.

---

As of today, the maintenance and usage burden is as follows:

1. We need to bump vunit/vhdl/osvvm and keep _add_osvvm() in sync with the exceptions (*_Aldec.vhd, *_c.vhd) in OSVVM.
2. We need to communicate which version of OSVVMLibraries do users need to download, in order to combine it with VUnit's OSVVM.
3. We need to provide an snippet such as https://github.com/umarcor/osvb/blob/94c13835db33f20893661cc627c94f1728197f77/AXI4Stream/test/osvvm/run.py and keep it in sync with the exceptions in the version of OSVVMLibraries matching VUnit's OSVVM.

We can do better, in the sense of reducing our maintenance effort, and making it easier for users to consume VUnit and OSVVMLibraries together.

Option 1: let VUnit submodule OSVVMLibraries instead of OSVVM (avoiding 2). We can keep building OSVVMLibraries/osvvm alone in _add_osvvm() and maybe provide an additional _add_osvvm_libraries() (which would avoid 3). The advantage of this approach is the ease of transition from the current codebase (just change the submodule and add a function to VUnit's API). It would also make the VUnit-OSVVM collaboration very explicit. However, the disadvantage is scalability: we cannot follow this approach with all third-party projects, nor recommend it as a general solution for companies to extend VUnit with their own libraries.

Option 2: let VUnit check if library OSVVM was declared/added before executing add_verification_components. Users might use different approaches to actually install OSVVM:

• Use OSVVM's .pro files (as reported by the OP), or GHDL's vendor scripts; then declare the libraries as external in VUnit.
• Use add_library and add_source_files to declare all the OSVVM libraries retrieved/downloaded to any location. That is: https://github.com/umarcor/osvb/blob/6958c2281044d494bfa543069f2b9f1f8c1f089d/AXI4Stream/test/osvvm/run.py.

Then, add_verification_components would be executed, which would not build the internal OSVVM, because it can see that the library name exists (either regular or external). The main advantage of this approach is that users can install any version of OSVVM they want; they can reuse a pre-compiled version, or they can build it per-script (as it is usual for VUnit's builtins). The main disadvantage is the same: users can install any version of OSVVM, which might not be compatible. However, I don't think this is critical, because VUnit does not have any strong dependency on any weird API feature of OSVVM; i.e. almost any OSVVM version should be valid for the random features required in VUnit.

From a technical point of view, the content of the run.py in Option 2 would be the same as the content of _add_osvvm() and _add_osvvm_libraries() in Option 1. That means we can have Option 2 implemented as a VUnit example. That would keep the snippet close to the VUnit repo, but not part of vunit_hdl codebase.

Option 3: the same as Option 2, but keeping the run.py snippet somewhere else (say OSVB or OSVVM), instead of the VUnit repo.

---

Overall, note that this is a particular case because VUnit depends on some OSVVM features, but not all of them. Therefore, we need to have a fallback solution that builds OSVVM's core automatically for less experienced users, while we want to allow flexibility for intermediate and advanced users. That's why I think that Options 2 or 3 are desirable.

I can help testing the approaches in ghdl/extended-tests, here and in OSVVM's CI. Each of them would test a different solution.

/cc @JimLewis @Paebbels

[JimLewis]

You could include OsvvmLibraries as a submodule. Can python execute foreign languages such as TCL? If so, then you could use the OSVVM compile scripts to build the libraries - although currently we are not auto-detecting Synopsys VCS, Cadence Xcelium, or Xilinx XSIM - so you would need to invoke them through their vendor specific startup.

Note I am not saying you should do this, but if you did do this, I would be happy to collaborate and make adjustments so things come together.

[umarcor]

@JimLewis (re)using the .pro (TCL) scripts somehow is something we are discussing in #765, and something I'm working on with @Paebbels. We want to use that in multiple projects (GHDL's vendor scripts, EDAA, pyVHDLModel, here...). We had a meeting about it yesterday. We are not convinced about calling TCL from Python, despite it being technically possible. Nonetheless, we want to propose some syntax enhancements to the .pro files, so that it is easier for us to parse/use them from Python without requiring a TCL interpreter.

This specific issue is about the conflicts because VUnit does have one version of OSVVM, which is a dependency of some of VUnit's libraries; and at the same time some users might want to use a different version of OSVVM. It is unrelated to how OSVVM libraries are built (OSVVM's scripts, GHDL's scripts or a plain VUnit run file); this is rather about what is done first and how to skip an internal step in VUnit's builtins management.

Note I am not saying you should do this

I do not have any objection to Option 1 (submoduling OSVVMLibraries instead of just OSVVM). If @LarsAsplund agrees with it, I'd be happy to see that collaboration. However, from a technical point of view, I believe it's better if users can retrieve and use any version of OSVVMLibraries. Your release frequency increased significantly since VUnit included OSVVM several years ago. Nowadays, we are not keeping up with your rythm. If we were to follow this approach, we would need 4-5 of us to collaborate in maintaining these 50 lines up to date: https://github.com/VUnit/vunit/blob/da8eb0aaa68d79a4d23f29b1a1053e0114b52ce5/vunit/builtins.py#L133-L178.

• You would provide the knowledge about which files need to be used/skipped.
• @Paebbels would provide the knowledge about dealing with .pro files from Python in different projects.
• @LarsAsplund would provide the knowledge about the VUnit API.
• I would keep coordination alive and handle CI/testing.
• Naturally, Tristan would be in the background providing GHDL.

This collaboration scheme is interesting for other areas as well: report merging, passing a runner generic to OSVVM tests, etc.

but if you did do this, I would be happy to collaborate and make adjustments so things come together.

That's so nice to hear! :heart: There are several tasks we can collaborate on so that VUnit users can more easily install and use OSVVMLibraries (using either of the three installation approaches). However, those are related to how OSVVM uses VHDL, and the nature of the methodology, rather than the build procedure itself. When we decide the solution to this issue and #754, I'll create CI jobs for testing. The matrix has 3x3=9 possible setups:

• Without VUnit's VCs/random, with VCs before adding OSVVMLibraries, or with VCs after adding OSVVMLibraries.
• Build with .pro files, with GHDL's vendor scripts, or with VUnit.

So, I'm creating the following tests in OSVB:

• test_AXI4Stream_OSVVM_VUnit
• test_AXI4Stream_OSVVM_VUnit_WithVCsBefore
• test_AXI4Stream_OSVVM_VUnit_WithVCsAfter
• test_AXI4Stream_OSVVM_VUnit_external_VendorScripts
• test_AXI4Stream_OSVVM_VUnit_external_VendorScripts_WithVCsBefore
• test_AXI4Stream_OSVVM_VUnit_external_VendorScripts_WithVCsAfter
• test_AXI4Stream_OSVVM_VUnit_external_ProFiles
• test_AXI4Stream_OSVVM_VUnit_external_ProFiles_WithVCsBefore
• test_AXI4Stream_OSVVM_VUnit_external_ProFiles_WithVCsAfter

NOTE: "Before" means using the OSVVM shipped with VUnit plus OSVVMLibraries retrieved "externally". Conversely, "After" means ignoring the OSVVM shipped with VUnit and using all externally retrieved OSVVMLibraries.

[LarsAsplund]

@umarcor @JimLewis I would go with option 2 because we shouldn't create more dependencies than needed and OSVVM is the only real dependency, All other libraries, whether they are part of OSVVMLibraries, UVVM, UVM or something else, should be possible to use together with VUnit although they are not shipped together. That's why #754 needs to be fixed.

I would also consider checking for new OSVVM releases in our CI as a reminder to bump to the latest version

[umarcor]

I added 10 run*.py scripts to OSVB:

• https://github.com/umarcor/osvb/tree/main/AXI4Stream/test/osvvm
• https://umarcor.github.io/osvb/egs/axi4stream.html#osvvm-tbstream

Those are all equivalent. The target testbench is TbStream.vhd + TestCtrl_e.vhd + TbStream_SendGet1.vhd, which is an example from OSVVM's AXI4 repository. All the variants are about different procedures for building OSVVM, OSVVMLibraries and/or VUnit's VCs, all together.

Currently:

• run.py, run_wvcsb and run_wvcsans are successful.
• run_wvcsa is to be fixed by #771.
• run_ext_* are not implemented yet; they are blocked by #754 and by GHDL's shell scripts not supporting OSVVMLibraries.

Apart from all these VUnit based Python scripts, the same subdir contains a run.pro TCL file and a run.sh shell script. The TCL file is the canonical build and simulation procedure provided by OSVVM's scripting plumbing. Conversely, the shell script uses GHDL's vendor scripts for compiling OSVVM. Running the simulation from the shell script is not implemented yet.

All the scripts are tested in CI: https://github.com/umarcor/osvb/actions/workflows/Test.yml

I would also consider checking for new OSVVM releases in our CI as a reminder to bump to the latest version

@LarsAsplund https://dependabot.com/ was an external service, which GitHub bought some months ago: https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/about-dependabot-version-updates. Adding a yml file should suffice for having PRs automatically created by dependabot. I use it several repos, such as https://github.com/im-tomu/fomu-workshop and https://github.com/ghdl/setup-ghdl-ci/pulls?q=is%3Apr+is%3Aclosed. dependabot supports git submodules as a packaging/distribution solution. However, I'm not sure it can handle OSVVM's tags, because they are not semver. I know we can get notified per commit/push, but I believe that's not something we want.

[LarsAsplund]

@umarcor Are you doing option 2 or 3 now?

depandabot looks nice but if it doesn't support non-semver versioning we can always write a script manually that handles OSVVM's year.month[patch]

[umarcor]

@LarsAsplund, it's technically Option 3, because the run*.py scripts are in OSVB, not here. However, I might add an example to this repo, based on run_wvcsa, which retrieves OSVVMLibraries. Let me know if you want to include that example in #771.

[LarsAsplund]

@umarcor Well, it doesn't really matter that much. If it's already in OSVB we better keep that as the single source. As long as we have an example we can direct people to.

[umarcor]

depandabot looks nice but if it doesn't support non-semver versioning we can always write a script manually that handles OSVVM's year.month[patch]

It seems that dependabot does not support the functionality we need: https://github.com/dependabot/dependabot-core/issues/1639. We'd better write our own script. I believe it should check all the submodules (OSVVM and JSON-for-VHDL).

@Paebbels, does https://github.com/paebbels/pyversioning allow to parse/get data from submodules?

[Paebbels]

The intension of pyVersioning is to extract version and meta information from:

• a config script
• current environment
• CI environments
• ...

to provide variables and templates, so these variables can be "printed" into files. One example is to gather:

• build date/time
• compiler
• project name, project variant
• git hash, git commit date/time, branch/tag, repository URL, ..

to translate it into a C file, so it can be compiled and linked into a project. Another usecase is to write information to a VHDL oder *.mem file to it can be translated into a read-only AXI4-Lite register. In case of an Xilinx MPSoC, software can now print version information of the used PL firmware and even reject execution on a PL that doesn't offer matching interfaces based on the version number (semantic version).

In addition, it is planned to also search submodules and other dependencies and integrate these information into the C or VHDL files. This would allow to see (and print) a Xilinx MPSoC what IP cores are used in what version/hash, ...

In general, somewhen such information would be available and we could think about using that data for other purposes.

---

So I think, pyVersioning is currently not offering what you ask for, but it would be in the scope.

[Paebbels]

@LarsAsplund The pyEDAA.projectModel can currently read *.xpr files and I have a prototype to read and parse *.pro files from OSVVM. It offers a general model for a "Project", "FileSet" and "File", which can be used e.g. by VUnit to read the list of files and call an add method on the VUnit classes for copying information from one representation to the other.

[umarcor]

FTR, in https://github.com/VUnit/vunit/issues/946#issuecomment-1636798612 @Blebowski reported using VUnit and NVC with OSVVM precompiled with NVC.

In the context of this issue, that's equivalent to the "GHDL's vendor scripts" solution.