search

VUnit / vunit

VUnit is a unit testing framework for VHDL/SystemVerilog
http://vunit.github.io/
Other
721 stars 258 forks source link

VCSMX Integration and exit status for failing tests #637

Open sbhutada opened 4 years ago

sbhutada commented 4 years ago

I have been able to integrate VCSMX into the VUNIT infrastructure.

I updated o sim_if/vcsmx.py o sim_if/factory.py o vcsmx_setup_file.py

When I run the failing testcase vunit infrastructure is marking it as pass – Any suggestions?

python run.py -v "lib.tb_example.Test that a failing test case actually fails" … EXIT STATUS= 2 Raising NonZeroExitCode= 2 Command Failed test starts = ['Test that a failing test case actually fails'] test_suite_done = True done = True test_name= Test that a failing test case actually fails result= TestStatus('passed') results= {'Test that a failing test case actually fails': TestStatus('passed')} results file = /global/gtsnaw_rwest4/bhutada/vunit-master/examples/verilog/user_guide/vunit_out/test_output/lib.tb_example.Test_that_a_failing_test_case_actually_fails_e0b95858e14bbfb897b8d111a43505ca7f742d28/vunit_results {'Test that a failing test case actually fails': TestStatus('passed')} sim_ok = False {'Test that a failing test case actually fails': TestStatus('passed')} False

cmarqu commented 4 years ago

Related: #134

sbhutada commented 4 years ago

Hi Colin, @cmarqu , @LarsAsplund

Attached is my updates to the vunit. I have added updated o sim_if/vcsmx.py o sim_if/factory.py o vcsmx_setup_file.py I have also added some print diagnostics to o suites.py o ostools.py

Couple questions:

To reproduce:

python run.py -v "lib.tb_example.Test that a failing test case actually fails" … EXIT STATUS= 2 Raising NonZeroExitCode= 2 Command Failed test starts = ['Test that a failing test case actually fails'] test_suite_done = True done = True test_name= Test that a failing test case actually fails result= TestStatus('passed') results= {'Test that a failing test case actually fails': TestStatus('passed')} results file = /global/gtsnaw_rwest4/bhutada/vunit-master/examples/verilog/user_guide/vunit_out/test_output/lib.tb_example.Test_that_a_failing_test_case_actually_fails_e0b95858e14bbfb897b8d111a43505ca7f742d28/vunit_results {'Test that a failing test case actually fails': TestStatus('passed')} sim_ok = False {'Test that a failing test case actually fails': TestStatus('passed')} False

Thanks, --Shashi

vunit_vcsmx.zip

sbhutada commented 4 years ago

The current approach is to change $error to $fatal (For Modelsim you guys are doing something that we cannot do for VCS):

BTW: GIT editor doesn't seem to work well with back tick, Verilog macros.

Any observations? Suggestions? Thanks.

LarsAsplund commented 4 years ago

I think that is an acceptable approach to move forward but I think you should open a ticket with Synopsys to get it fixed in future releases. A key feature of VUnit is that it should work with legacy testbenches using $error and not our CHECKs.

sbhutada commented 4 years ago

$error is only meant to print out errors. $fatal is to exit the sim. For timeout why not change the $error to $fatal. With ModelSim you are doing something TCL that I am not clear on, and not sure if we can do it in other simulators like VCS.

LarsAsplund commented 4 years ago

The basic approach to error detection in VUnit is

The exception is our VHDL checks that have internal error counters that are checked when the simulation ends. This means that we can configure it to not stop on error and still detect the errors on simulation exit and mark the test as failed

This approach allow us to detect Verilog $error, VHDL assert, division by zero, null pointer dereferencing, PSL errors etc. This is different from other test frameworks like UVM, OSVVM and UVVM which requires that you use their own error mechanisms (for example uvm_error) to have errors detected.

The principle is that you should be able to use VUnit with a legacy testbench using standard language error mechanisms. You should not have to rewrite your code and you should not have to parse the log for error messages. Here is an example.

sbhutada commented 4 years ago

For some reason - with VCS - even after $error is invoked correctly and sim exit status is set to 2 - meaning failure - the vunit infrastructure (attached earlier here) seems to detect it as a test passed. My guess is it has to do with cleanup - that adds the "test_suite_done" string in "vunit_results". When I look at the cleanup code in vunit_pkg.sv cleanup code it is not checking if there was any error. With Modelsim it somehow hijacks the sim to proceed any further based on parsing of that file in TCL. Not all simulators will have that feature in TCL.

Any other approach or suggestions? Thanks.

sbhutada commented 4 years ago

The VUNIT behavior for one of the failing test with VCS is as follows ...

I have added some ExitStatus/results/sim_ok vunit py diagnostics inside os/suites.py ... and you can see at the end how test_suit_done is messing up the sim_ok for a failing test ... How to stop the vunit_pkg::cleanup() code from adding "test_suite_done" flag into the vunit_results file for test that has error exit status?


...
UVM_INFO @ 0: reporter [RNTST] Running test test_failing_with_uvm_error...
UVM_ERROR /global/gtsnaw_rwest4/bhutada/vunit-master/examples/verilog/uvm/testuvm.sv(29) @ 0: uvm_test_top [] Something bad happened
UVM_INFO /global/snps_apps/vcs_2020.03-Beta/etc/uvm-1.2/base/uvm_objection.svh(1276) @ 0: reporter [TEST_DONE] 'run' phase is ready to proceed to the 'extract' phase
UVM_INFO /global/snps_apps/vcs_2020.03-Beta/etc/uvm-1.2/base/uvm_report_catcher.svh(705) @ 0: reporter [UVM/REPORT/CATCHER]
--- UVM Report catcher Summary ---

Number of demoted UVM_FATAL reports  :    0
Number of demoted UVM_ERROR reports  :    0
Number of demoted UVM_WARNING reports:    0
Number of caught UVM_FATAL reports   :    0
Number of caught UVM_ERROR reports   :    0
Number of caught UVM_WARNING reports :    0

UVM_INFO /global/snps_apps/vcs_2020.03-Beta/etc/uvm-1.2/base/uvm_report_server.svh(894) @ 0: reporter [UVM/REPORT/SERVER]
--- UVM Report Summary ---

** Report counts by severity
UVM_INFO :    4
UVM_WARNING :    0
UVM_ERROR :    1
UVM_FATAL :    0
** Report counts by id
[]     1
[RNTST]     1
[TEST_DONE]     1
[UVM/RELNOTES]     1
[UVM/REPORT/CATCHER]     1

"/global/gtsnaw_rwest4/bhutada/vunit-master/examples/verilog/uvm/testuvm.sv", 99: tb_failing.unnamed$$_1.unnamed$$_2: started at 0fs failed at 0fs
        Offending '(server.get_severity_count(UVM_ERROR) === 0)'
Error: "/global/gtsnaw_rwest4/bhutada/vunit-master/examples/verilog/uvm/testuvm.sv", 99: tb_failing.unnamed$$_1.unnamed$$_2: at time 0 fs
CHECK_EQUAL failed! Got server.get_severity_count(UVM_ERROR)=1 0 0.
vunit_pkg.sv, 138 :       $stop(0);
ucli% quit
           V C S   S i m u l a t i o n   R e p o r t
Time: 0 fs

**EXIT STATUS= 2
Raising NonZeroExitCode= 2
Command Failed**
test starts = ['Test that your CI fails on UVM error']
**test_suite_done = True**
done = True
test_name= Test that your CI fails on UVM error
result= TestStatus('passed')
results= {'Test that your CI fails on UVM error': TestStatus('passed')}
results file = /global/gtsnaw_rwest4/bhutada/vunit-master/examples/verilog/uvm/vunit_out/test_output/lib.tb_failing.Test_that_your_CI_fails_on_UVM_error_2691d2f255c680502cf04ae97171972035dee316/vunit_results {'Test that your CI fails on UVM error': TestStatus('passed')}
sim_ok = False {'Test that your CI fails on UVM error': TestStatus('passed')} False