VUnit / vunit

VUnit is a unit testing framework for VHDL/SystemVerilog
http://vunit.github.io/
Other
728 stars 262 forks source link

Support for Cadence NCSim? #92

Closed LarsAsplund closed 7 years ago

LarsAsplund commented 8 years ago

Recently I had a question if it would be possible to support NCSim from Cadence. Before looking more into this I'd like to see if there are others out there looking for this support. If so, comment this question.

mark-newsam commented 8 years ago

I would be interested in this support

LarsAsplund commented 8 years ago

Hi Mark

I guess that you're using NCSim yourself. Any connections with Cadence that would give us an evaluation license to make this happen? In the long run we would like some sort of tool provider license to enable continuous testing with NCSim but an evaluation is a good starting point such that we can deliver a proof of concept and hopefully show for Cadence that there is genuine interest among NCSim users to have this support in the future as well. You can contact me at lars.anders.asplund@gmail.com if you prefer to have that discussion offline. If you know other NCSim users that would like to see VUnit support as well I'd appreciate if you would direct them here and make their voices heard.

mark-newsam commented 8 years ago

Hi Lars, We use both NCSim and Modelsim in our company I mainly use Modelsim but was looking at your software as we are starting to use Jenkins and you have support for it. Unfortunately I don't have much contact with Cadence so not sure I can help with an evaluation license. I could try out the odd build on NCSim if that helps.

LarsAsplund commented 8 years ago

Thanks Mark,

Porting to a new simulator has not been very difficult so far but it requires that we create a new Python interface class which means that we'll need an evaluation license to do that efficiently. It usually involves some work on the VHDL packages as well due to simulator bugs and differences in the interpretation of the VHDL standards. If we succeed with that but do not get a tool provider license it would be interesting to have users that can verify new releases although it normally works for all simulators once you make it work for one. From what I understand NCSim do not support VHDL standards newer than VHDL-93. The simulators we support so far all support VHDL-2008 but we verify with VHDL-93 and VHDL-2002 as well for ModelSim. There are some packages in VUnit which are not part of the "core" that only work with VHDL-2008 (array, com, and the redistribution of OSVVM) but these are utility packages and not necessary for (unit) testing. One thing you could do is to look at the compile order using ModelSim and then compile the same files for NCSim and see what happens. You can find the compile order if you go to vunit/vhdl and

> set VUNIT_VHDL_STANDARD=93
> python compile_vunit_lib.py --clean
mark-newsam commented 8 years ago

Hi Lars,

NCSim (Now called Incisive by Cadence) does support 2008 although a little selective.

It was a bit of a mission but I can compile the vunit_lib with only a couple of issues.

  1. Don't seem to have support for "context" keyword
  2. Function "srl" in check.vhd, removing it I was able to compile the file.

To compile I used the following: irun -prep -compile +overwrite -64bit -extv200x -v200x -nclibdirname vunit_out/cadence/libraries \ -makelib vunit_lib -F ./vunit_lib.vc -endlib

Where the file vunit_lib.vc contains: ./vunit/vhdl/string_ops/src/string_ops.vhd ./vunit/vhdl/run/src/stop_api.vhd ./vunit/vhdl/run/src/stop_body_2008.vhd ./vunit/vhdl/vhdl/src/lang/lang.vhd ./vunit/vhdl/path/src/path.vhd ./vunit/vhdl/vhdl/src/lib/std/textio.vhd ./vunit/vhdl/logging/src/log_types.vhd ./vunit/vhdl/logging/src/log_formatting.vhd ./vunit/vhdl/logging/src/log_special_types200x.vhd ./vunit/vhdl/logging/src/log_base_api.vhd ./vunit/vhdl/logging/src/log_api.vhd ./vunit/vhdl/check/src/check_types.vhd ./vunit/vhdl/check/src/check_special_types200x.vhd ./vunit/vhdl/check/src/check_base_api.vhd ./vunit/vhdl/check/src/check_base.vhd ./vunit/vhdl/check/src/check_api.vhd ./vunit/vhdl/dictionary/src/dictionary.vhd ./vunit/vhdl/run/src/run_types.vhd ./vunit/vhdl/run/src/run_special_types200x.vhd ./vunit/vhdl/run/src/run_base_api.vhd ./vunit/vhdl/run/src/run_base.vhd ./vunit/vhdl/run/src/run_api.vhd

./vunit/vhdl/vunit_context.vhd

./vunit/vhdl/run/src/run.vhd

./vunit/vhdl/vunit_run_context.vhd

./vunit/vhdl/logging/src/log_base.vhd ./vunit/vhdl/check/src/check.vhd ./vunit/vhdl/logging/src/log.vhd

NB. Two context files removed

kraigher commented 8 years ago

@mark-newsam We would really need to have a license for NCSim ourselves to be able to add official support for it. Without the simulator at hand to experiment and no documentation it is not really feasible to attempt it. There are many subtle things to get wrong without trying it out for real.

We have a set of acceptance tests that we want to be able to work on a given simulator back-end for us to consider the simulator working. This is to ensure that test that shall pass will pass and that tests that shall fail will fail.

A NCSim user experienced with Python could make a Fork and create unofficial NCSim-support. A simulator back-end is just a class with certain methods. ghdl_interface.py could be used as a starting point. We could support such an attempt.

LarsAsplund commented 8 years ago

@mark-newsam Strange that context isn't supported. It's very useful and doesn't seem to be something that would be hard to support.

What was the error message for srl? Does it work if you make that a function instead?

Anyway, I found a tool provider alliance program with Cadence and I will contact them to see if that is something that would help us with the license we need. This is a promising start though.

cmarqu commented 8 years ago

Please consider me interested.

cmarqu commented 8 years ago

@LarsAsplund I can answer the questions for @mark-newsam:

        ././vunit/vhdl/check/src/check.vhd:
          function "srl" (
                       |
        ncvhdl_p: *E,SPBCFM (././vunit/vhdl/check/src/check.vhd,21|15): subprogram body does not conform with earlier declaration [2.2] [2.7].
                errors: 1, warnings: 0

With a function, I get:

          function srl (
                   |
        ncvhdl_p: *E,EXPFUD (././vunit/vhdl/check/src/check.vhd,21|11): expecting a function designator [2.1].
          end function srl;
                       |
        ncvhdl_p: *E,EXPSMI (././vunit/vhdl/check/src/check.vhd,31|15): expecting a semicolon (';') [2.2].
                errors: 2, warnings: 0

It works when I use an extended identifier: function \srl\.

This is with IUS 14.20-s014.

LarsAsplund commented 8 years ago

Thanks @cmarqu. It seems like overloading of srl doesn't work and when you make a normal function it fails since srl is a reserved word. That would explain why an extended identifier works. It might still be my fault though. I just realized that standard srl operators take an integer as the second operand so maybe it doesn't consider this as an overload attempt. Could you test changing natural with integer? If that doesn't work I will just create a function with another name.

cmarqu commented 8 years ago

Unfortunately, replacing natural with integer doesn't work either.

LarsAsplund commented 8 years ago

@cmarqu Ok, it was worth a try. I will just go ahead and create a function with a non-reserved identifier. Next step would be to try to get that license. Hopefully the interest expressed here and offline is sufficient. I'll keep you posted.

cmarqu commented 8 years ago

It looks like the SystemVerilog flavour from #49 works in Cadence NCSim (or rather call it "Cadence Incisive"). Maybe we could already add an incisive_interface.py to allow for easier experimentation? Actually, I have already played with it a bit, but I'm unsure about the approach. Incisive has two ways of starting a simulation: the "legacy" three-step way consisting of "ncvlog/ncvhdl"+"ncelab"+"ncsim"; and the recommended way with "irun" which you can then restrict to only compile or compile+elaborate with command line switches. Does VUnit ever need these intermediate compile and elaborate steps, or does it care only about running a simulation?

kraigher commented 8 years ago

@cmarqu If you are interested in getting Cadence Incisive to run by helping create an incisive_interface.py I could support you. You could have a look at ghdl_interface.py since it is the least complex simulator interface we have with less other noise for you to cipher through.

VUnit has a incremental compilation which means that if you have 100 files and you change 1 it only re-compiles the 4 files that were effected. This should be implemented by the compile_project method of the simulator interface class. The project object will tell you what you need to re-compile.

The simulate method of the simulator interface class relies on that the compile_project was previously run. Its main role is to simulate the top level specificed. We do have an --elaborate flag which where the design is only elaborated and that is currently implemented by the simulate method. So currently the simulate method can do either elaborate + simulate or just simulate. If this does not map well to Incisive I would advice you to ignore the elaborate_only configuration for now. We can change this structure to fit Incisive if the other parts prove successful.

cmarqu commented 8 years ago

@kraigher Thanks! I have indeed started by taking ghdl_interface.py because you mentioned it somewhere else before. Incremental compilation is also handled by irun internally, but it's probably easier to not rely on that and just follow the way the other tool support does it. Let me work on this a little bit more and get back to you with more specific questions if needed.

A bit unrelated: unless I added lib.add_source_files("../../../vunit/verilog/*.sv") to examples/verilog/user_guide/run.py, the simulator didn't know about vunit_pkg (because it hasn't seen vunit_pkg.sv). How is vunit_pkg meant to be known to the simulator? Precompiled before? But then I'm missing the place where the external library is referenced...?

kraigher commented 8 years ago

@cmarqu If you want to start experimenting you could just do a GitHub Fork of VUnit. Thus your efforts would be visible to us and we can comment on them and support you.

Well SystemVerilog support is still experimental... vunit_pkg.sv is compiled into vunit_lib which is different from tb_uart_lib. SystemVerilog does not really have good support for libraries in the same way as VHDL does. For ModelSim we explicitly make Verilog all libraries visible to one another. You would have to do the same for Incisive if it is possible.

Maybe you are only interested in SystemVerilog but when porting VUnit to a new simulator I would first start with the VHDL-only part.

kraigher commented 8 years ago
PYTHONPATH=. python examples/verilog/uart/run.py 
Compiling /home/kraigher/repo/vunit/examples/verilog/uart/src/uart_rx.sv into uart_lib ...
Compiling /home/kraigher/repo/vunit/examples/verilog/uart/src/uart_tx.sv into uart_lib ...
Compiling /home/kraigher/repo/vunit/vunit/verilog/vunit_pkg.sv into vunit_lib ...
Compiling /home/kraigher/repo/vunit/examples/verilog/uart/src/test/tb_uart_tx.sv into tb_uart_lib ...
Compiling /home/kraigher/repo/vunit/examples/verilog/uart/src/test/tb_uart_rx.sv into tb_uart_lib ...
Starting tb_uart_lib.tb_uart_rx.test_tvalid_low_at_start
pass (P=1 S=0 F=0 T=5) tb_uart_lib.tb_uart_rx.test_tvalid_low_at_start (0.9 seconds)

Starting tb_uart_lib.tb_uart_rx.test_receives_one_byte
pass (P=2 S=0 F=0 T=5) tb_uart_lib.tb_uart_rx.test_receives_one_byte (0.2 seconds)

Starting tb_uart_lib.tb_uart_rx.test_two_bytes_cause_overflow
pass (P=3 S=0 F=0 T=5) tb_uart_lib.tb_uart_rx.test_two_bytes_cause_overflow (0.2 seconds)

Starting tb_uart_lib.tb_uart_tx.test_send_one_byte
pass (P=4 S=0 F=0 T=5) tb_uart_lib.tb_uart_tx.test_send_one_byte (0.2 seconds)

Starting tb_uart_lib.tb_uart_tx.test_send_many_bytes
pass (P=5 S=0 F=0 T=5) tb_uart_lib.tb_uart_tx.test_send_many_bytes (0.3 seconds)

==== Summary ================================================================
pass tb_uart_lib.tb_uart_rx.test_tvalid_low_at_start      (0.9 seconds)
pass tb_uart_lib.tb_uart_rx.test_receives_one_byte        (0.2 seconds)
pass tb_uart_lib.tb_uart_rx.test_two_bytes_cause_overflow (0.2 seconds)
pass tb_uart_lib.tb_uart_tx.test_send_one_byte            (0.2 seconds)
pass tb_uart_lib.tb_uart_tx.test_send_many_bytes          (0.3 seconds)
=============================================================================
pass 5 of 5
=============================================================================
Total time was 1.9 seconds
Elapsed time was 2.2 seconds
=============================================================================
All passed!
kraigher commented 8 years ago

Since SystemVerilog support is still experimental one future outcome could be that we do not have things in different libraries for SystemVerilog like we do with VHDL but I would like to avoid it if possible.

LarsAsplund commented 8 years ago

@cmarqu I replaced the srl overloading with a function call. Running the 2002 version of VUnit without any contexts should hopefully work now.

cmarqu commented 8 years ago

I have started my fork at https://github.com/cmarqu/vunit. Some things still need work though:

And then something still prevents me from running the simulation successfully - in VHDL, it's the context stuff, in SystemVerilog it's the library visibility. I know that I could make it work by introducing simulator-specific hacks, but that defeats the purpose... Have you guys considered setting up a Gitter chat channel for VUnit for faster turnaround times?

LarsAsplund commented 8 years ago

@cmarqu Have you tried compiling for VHDL 2002? Then there will be no contexts. Contexts are still used in many examples though but if you just want to see if those examples work you can simply replace those lines with the use statements find in the context file. In most cases you wouldn't need all those use statements but that's a simple start.

kraigher commented 8 years ago

We can add hooks to declare that a simulator does not support contexts. I will add this in the evening when I am not working. The drawback of using 2002 is that there is no good portable stop mechanism as for 2008. Only Modelsim supports non 2008 because of this where an internal signal is used to stop. We migth change the non 2008 stop mechanism to be asser fatal with special message instead.

kraigher commented 8 years ago

I will look into Gitter as well.

LarsAsplund commented 8 years ago

@kraigher I think I already set it up but I haven't found out how to use it yet.

kraigher commented 8 years ago

I usually work during the day and cannot spend any significant time to support then. I can work on VUnit on early mornings and evenings as well as weekends.

cmarqu commented 8 years ago

Incisive doesn't seem to have a way to explicitly select VHDL-2002, it just knows the "-v200x" and "-extv200x" switches.

cmarqu commented 8 years ago

Gitter doesn't need to be a fulltime support channel, I'm sure there will be some overlap in the times when we both/three can work on VUnit. https://gitter.im/home/createroom - it offers your own Repositories when you select the second option.

kraigher commented 8 years ago

I would configure incisive to use as much new VHDL as possible always and work from there. There is no point running the simulator more restrictive than it can be. The standards are backwards compatible. Just make the hacks you need to make it work and I will help you make them in a clean way by adding hooks.

LarsAsplund commented 8 years ago

@kraigher, @cmarqu There is now a VUnit room

cmarqu commented 8 years ago

@kraigher Ok, I'll wait for "We can add hooks to declare that a simulator does not support contexts" and go from there with any further hacks.

mark-newsam commented 8 years ago

I had a reply from Cadence on the lack of context support:

R&D has put this CCR status as inactive backlog and will implement this in future release. But currently there is not any immediate plan for this feature implementation. Please let me know if you have any questions, concerns?

Since the CCR associated with this Case is in inactive/backlog state which is a final state, so there is no action required between us on this issue. So, I’ll be going ahead and close this case for now. Hope that is fine with you ?

Getting 2008 support from Cadence is going to be rather slow...

cmarqu commented 8 years ago

@mark-newsam I have gotten a similar response. Let's hope that with growing popularity of VUnit and similar packages, they will feel some pressure.

kraigher commented 8 years ago

It really should not be hard for them to add context support. Both GHDL and NVC open source simulators added it with literally only hours worth of efforts. It is really nothing more than a convenient way to group use-clauses together that should be fairly straightforward to implement in any simulator code base.

kraigher commented 8 years ago

@cmarqu I have pushed an incisive branch which you can rebase on to get support for avoiding contexts.

LarsAsplund commented 8 years ago

@mark-newsam @kraigher @cmarqu Yes, the lack of context support is a bit strange given how useful it is.

LarsAsplund commented 8 years ago

@mark-newsam Great progress is made on https://gitter.im/VUnit/vunit. Please join the chat.

cmarqu commented 8 years ago

The code is now fully in https://github.com/VUnit/vunit/tree/incisive - I have deleted my fork.

leoheck commented 8 years ago

Hi, the interface for cadence is already working? How can I test it? I tried today at my work but the examples didn't work out of the box. Anyway, I can give some help to test it.

cmarqu commented 8 years ago

@leoheck Did you get the incisive branch, and have irun in your $PATH? It should work in principle, just some of the acceptance test fail because of internal errors in the simulator. I had reported these (two or three) to Cadence while at my old company, but as far as I know these tickets are still open (someone else is monitoring them for me). And of course, the OSVVM code doesn't compile in Incisive yet, see https://github.com/JimLewis/OSVVM/issues/7.

leoheck commented 8 years ago

Hm, good to know. I had the incisive on PATH and I just thought that run.py would use the first available tool it finds. Good to know that I have to switch to the incisive branch. Thank you.

LarsAsplund commented 7 years ago

I think we should close this one. I'm expecting more issues as long as the support is experimental but I think such issues should be more specific.