Ability to call SystemVerilog task

[Alex-Mann]

I'm wondering if there is any ability to call a SystemVerilog/UVM Task/Function from Cocotb.

Instead of living in purely Cocotb, or purely UVM, is it possible from the python side to direct and control the simulation? Say I start my test by calling a Cocotb async routine, then I want to call a SystemVerilog function that is exposed at the TB level? Maybe this is not an ideal use case, but I think it would provide a great amount of flexibility and help to drive even more adoption to Cocotb.

I know there is the DPI interface piece in SV, but having a really easy way to call one of these externs from the Cocotb python side would be awesome.

[ktbarrett]

1217 is related.

[ktbarrett]

To expand on the original inquiry about calling SystemVerilog tasks (or VHDL procedures) from cocotb the best to accomplish that is using the SystemVerilog-DPI for Verilog, or VHPIDIRECT in GHDL (if @umarcor get his way it will be a standard).

There is nothing inherently in the way of using these interfaces simultaneously with cocotb (or any VPI or VHPI cosim). For proper cocotb support though we need these the interface to tie into the simulator since tasks can consume simulation time. The easiest way to accomplish this is using cocotb.external. The interface functions could interact with the exported DPI/VHPIDIRECT functions using ctypes if the interface is simple enough, so it could be "easy".

I think we have all the necessary pieces to prototype this, if someone is up to it.

[umarcor]

AFAIU, the VHDL standard does not define how to execute VHDL functions procedures from a foreign language. There are two possibilities only:

• To execute foreign functions from VHDL (through VHPIDIRECT).
• To use VHPI.

Apart from that, GHDL allows VHDL functions/procedures to be executed from a foreign function. However, this is more of a side effect (because the internal architecture of GHDL is similar to a compiler) than a feature which is implemented on purpose. Hence, it is implementation dependent, not standarized.

I am trying to have the VHDL standard extended/enhanced, to include the details which are currently undefined. These are mostly:

• How should fat pointers look like, or whether tools should provide a header to decode/encode fat pointers.
• How should VHDL functions/procedures be exposed in C; what is the name of the symbol.
  • How to pass a context, so that VHDL functions/procedures executed from C can have wait statements.

This is not a new idea. A proposal exists in the standarization group since 2011: http://www.eda-twiki.org/cgi-bin/view.cgi/P1076/DpiProposal. My contribution is being to write down what I have gotten to learn about how GHDL's VHPIDIRECT works:

• https://github.com/umarcor/ghdl-cosim/blob/vhdl202x-lcd-dpi/vhdl202x-lcs-dpi/main.md
• https://ghdl.github.io/ghdl-cosim/

Other users in the VASG are interested on the feature precisely to allow a better integration of VHDL and SystemVerilog and/or Verilog. Hence, the standarized result is unlikely to be an exact copy of how GHDL works today, but I believe it is a solid foundation.

Regarding co-execution of VHDL and Python, the current problems with GHDL are:

• It is a little more verbose than I'd like to, because it seems that an intermediate C file is required. I hope that can be improved (both in VHDL/VHPIDIRECT and in Python/ctypes): ghdl/ghdl#1398.
• Failing simulations exit with an abortion that forces any wrapping Python to close abruptly: ghdl/ghdl-cosim#15. I believe this is a bug in GHDL, but Tristan is currently very busy with synthesis.

Apart from those points, it works and it is currently usable (albeit not completely documented).

Overall, it would be really nice if some developers/maintainers/users of cocotb took part in the discussion. VASG meets every ~2 weeks. There is also a reflector/mailing-list. Moreover, members and "usual suspects" in the VASG participate in many gitter rooms (VHDL, ghdl...). I think that ghdl/ghdl#1398 is a good first reference to understand what's currently possible and which is my proposal for the future.

[Alex-Mann]

I can take a crack at prototyping this, but it would be helpful if you could give a general overview of how and where you would structure the calls and integrations.

[eric-wieser]

The easiest way to accomplish this is using cocotb.external.

cocotb.external tasks are not allowed to advance the scheduler unless they go through cocotb.function to do so, which may cause an issue.

[ktbarrett]

@eric-wieser Yeah, started looking more into it, it is a problem. We need something that behaves like a trigger so we can allow the simulator to advance and re-enter the scheduler once complete.

[eric-wieser]

I don't really understand how systemverilog DPI is supposed to work. I assume the systemverilog first calls into the C code, and not the other way around?

[ktbarrett]

From the little I know about it, DPI is just an FFI, it doesn't define any control. You can write testbench HDL code that calls into an imported C function, or you can call into an exported SV task or function from a cosim application (or the aforementioned C helper function).

[eric-wieser]

My current understanding of DPI is:

• Calling a task from C results in your C thread being suspended
• Calling a C function from verilog spawns a new thread for your function
• The former can only be done if your thread originated from the latter

[Alex-Mann]

So I'll try to summarize what I've been reading about the DPI here incase it helps spur some new ideas on how to implement this...

As @ktbarrett said, it really is just an FFI for SV. It allows you to:

• import functions or tasks defined in C
• export functions or tasks defined in SV

The SystemVerilog DPI supports two types of functions:

Pure Functions

A nonvoid (must return a value) function which depends on only the input arguments and contains no side effects. They cannot:

• perform any file IO
• read or write anything (ie. no env variables, OS objects, shared memory, etc)
• access any persistent data like global or static variables

Context Functions

An imported C function which calls exported tasks or functions, or accesses SystemVerilog data objects other than its actual arguments (taken from Wikipedia). These functions can call the VPI.

---

Without knowing really how the C / Cpp layer of Cocotb works.. it seems like you would need to be aware of extern functions/tasks at compile time, and then provide a cocotb api to call this exported function from the python side since it would have to be able to advance simulation time and fit into the overall Cocotb model.

[ktbarrett]

@eric-wieser I've been reading over the spec [1] and it seems that any caller can call any exported task given they first change their context to the scope where the exported task is defined using svSetScope (I'm guessing the idea is that a particular scope is handled by a single thread where re-entrancy is safe?). They even specifically mention calling an exported task from a VPI callback in 35.5.3.

@Alex-Mann I don't think any C level changes are required of cocotb, at least not for a prototype.

I think this will require the following:

• export functions/tasks with DPI
  • not testing import functions/tasks as it's a whole other beast that's a lot more wicked
• create a shim library
  • this will do the necessary C things to interface with the DPI and provide a more regular interface to a caller
  • compile this as a shared library
• use the shim from cocotb
  • Use ctypes to load the library
  • create a Python wrapper for calling the C function using ctypes
  • wrap the Python function in something to make it appear to be a trigger to the scheduler

Most of this is out-of-scope for inclusion into cocotb, although we definitely want this documented. One thing cocotb could incorporate is the last point above. This 'thing' would be a Trigger, and it would call a Python function with a callback to send the trigger to scheduler.react once complete.

[1] You can find it here, section 35, page 939, if you don't have a IEEE membership on hand.

[eric-wieser]

From that same section:

The behavior of the DPI scope-related APIs and invocation of DPI export subroutines will be simulator defined and is beyond the scope of the DPI specification.

It sounds like svSetScope must either:

• Somehow change the thread of the caller, via setjmp / longjmp shenanigans
• Force the PLI event handler thread to jump ship to another thread.

Both of these sound likely to make cocotb unhappy.

[ktbarrett]

There is also an Annex H (starting on page 1222) that gives more detail on the DPI.

The behavior of DPI utility functions that manipulate context is undefined when they are invoked by any subroutine that is not part of a DPI context call chain (see 35.5.3)

Which implies what we'd like to do is undefined.

Section H.9.5 goes over the interaction between DPI and VPI, but completely fails to mention calling exported tasks from VPI callbacks. Not mentioning it usually means it's unsupported.

But from 35.5.3

A foreign language subroutine supported through some other interface (a VPI callback for example), can also make a call to svSetScope or to other DPI scope-related APIs. This foreign language subroutine can also call an export subroutine declared in a specific instantiated scope by first making a call to svSetScope. The behavior of the DPI scope-related APIs and invocation of DPI export subroutines will be simulator defined and is beyond the scope of the DPI specification.

Which is inconsistent with what was said in Annex H.

My best guess is they simply didn't expect people to be doing what we are trying to do and it's not well specified. And I doubt most simulators will support this use case gracefully. So I'm probably wrong, I don't think we can support this in combination with cocotb. DPI functionality can be supported in concert with cocotb only if the two are totally independent. I don't think signalling between the two would work terribly well either.

Pinging @hackfin, he was asking some questions related to this the other day (of course it was GHDL and VHPIDIRECT, but similar).

[marlonjames]

Also from 35.5.3:

A DPI import call chain is an inter-language call chain starting from SystemVerilog into a subroutine that is defined in a DPI supported language. The starting point of the call chain from SystemVerilog is called the root of the call chain.

So the DPI spec only defines how to start from SystemVerilog and go to C through an imported subroutine. Support for the root of the call chain being in a VPI callback is simulator dependent.

[github-actions[bot]]

Has your question been resolved? If so please close this issue. If it has not been resolved, you may need to provide more information. If no more activity on this issue occurs in 7 days, it will be closed.

[mballance]

Apologies for jumping into this thread late... I do see value in being able to call DPI functions/tasks from cocotb (either directly or in the context of a library). Depending on the simulator, cocotb may need to take some action to support this out of the box. Specifically, I believe a DPI-enabled GPI implementation would be required to ensure that cocotb runs within a DPI context. I think the actual code involved (on top of the VPI-based GPI implementation) would be fairly minimal. That said, it would be another GPI implementation. Is there openness to having another GPI backend?

[ktbarrett]

@mballance I'm not so sure how that would work? As stated in this issue, calling a DPI function/task from a VPI callback does not seem to be supported by the standard. We don't have anything against someone developing a DPI/VPI GPI implementation as an extension. Are you or @Alex-Mann willing to push the standards committee to clarify/fix the standard?

[mballance]

Hi @ktbarrett, the key issue that I've observed is that calls from cocotb back into the simulator via DPI come from VPI-land, and thus do not have a DPI context. If callbacks (sim-event, time-callback, signal-callback) are routed through DPI prior to calling cocotb, it should then be fine to make DPI calls from cocotb. Clarifying the standard would be a good long-term step. Of course the clarification could end up being that VPI->DPI calls are not supported.

[ktbarrett]

If callbacks (sim-event, time-callback, signal-callback) are routed through DPI prior to calling cocotb, it should then be fine to make DPI calls from cocotb.

How is that possible?

[mballance]

@ktbarrett, As they say, the devil is in the details... Some of the details will hinge on exactly what cocotb users that want to use DPI require in terms of simulation access. Is it sufficient to have access to simulation- and time-event callbacks, and use DPI for interacting with the design? Or, is full signal access required? By the way, I did dig up some interesting work that Jonathan Bromley did on connecting VPI and DPI. Possibly some directions to explore... https://www.verilab.com/files/snug_2012_presentation71_final.pdf https://www.verilab.com/files/snug_2012_paper71_final.pdf

My primary interest, though, was understanding the openness of the maintainers to having a new GPI implementation. I wasn't sure if the maintenance burden is sufficient that strongly capping the number is important.

[ktbarrett]

@mballance Maintenance burden is not an issue if the implementation is not incorporated into this repo. We are trying to avoid becoming a dumping pile of good ideas. We want to provide a solid core of minimum features for users to write testbenches in Python "the cocotb way" and provide good interfaces for extensions.

It's not currently possible to make a new GPI implementation as an extension (though it's not far off), but we would rather work on making that a possibility than maintaining another GPI implementation in-repo. The GPI certainly needs a second pass to redesign it as an end product and not as an implementation detail. There might even be others interested in such a thing.

[mballance]

@ktbarrett, makes sense and appreciate the feedback.

[Alex-Mann]

Is there an overview on the implementation/architecture of the GPI or the best thing to do to just dig into the source code?

[ktbarrett]

@Alex-Mann Unfortunately, digging into the source code and asking questions on the gitter is the best thing to do now. We do plan on documenting it all at some point here.

[ktbarrett]

Following on this discussion and those papers, it seems the best way of "calling" a DPI export is actually to not call the DPI export, but instead communicate over an asynchronous queue. The VPI callback would add "calls" to the queue (TLM queue?) and toggle a notification signal in the design, to signal that there are messages on the queue to consume. A SystemVerilog process would run when that notifier changes, pull messages from the queue by calling a DPI import, and does the actual task call with that data.

That is a serious amount of machinery that may be easy to automate.

[Alex-Mann]

When you refer to "those papers" which ones exactly are you talking about? I'd like to take a look through them if there is some new material/ideas.

[ktbarrett]

@Alex-Mann The ones that @mballance posted. In the second link on page 6 it goes over how it worked around the inability to call DPI exports from VPI callbacks. I simply worked from there with this specific task in mind.

[Alex-Mann]

Hi @mballance, I just watched your talk you gave at Latch Up (great stuff) - does this mean my ask in this thread could now be supported in Cocotb? Are there any limitations currently?

I would be interested in putting together a prototype of this which I could back contribute as an example to cocotb if yes.

[mballance]

Hi @Alex-Mann, yes, that is my understanding. It should now be possible to cross-call between Python+cocotb and SystemVerilog tasks. At the moment, known limitations are around platform (just Linux at the moment, though there are no technical issues supporting OSX and Windows) and testing across all simulators (development has been with one OSS and one closed-source simulator). My goal would be to have an example of cocotb/SV interaction in both the PyHDL-IF package and in cocotb. I think there's the beginnings of one in the PyHDL-IF package, but let me confirm. I'll give an update on this thread, and you can use that example as a starting point for a cocotb-contributed example.

[mballance]

Wanted to provide a bit of an update. There is now a first example showing cocotb and PyHDL-IF working together here: https://github.com/fvutils/pyhdl-if/tree/main/examples/call/cocotb/call_sv_bfm

The basic concept is using cocotb for some signal-level interations (wait for reset), while calling the tasks of a BFM implemented directly in SV.