Questa/Qrun simulator does not work for first run for full regression tests. It works when re-run with --noclean switch

danghai commented 4 years ago

Hi @taoliug . I got a lot of failure for running the test. One of them is riscv_rand_instr_test The error log file:

# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_jal_instr ratio:20/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/test/riscv_instr_base_test.sv(99) @ 0: uvm_test_top [uvm_test_top] All directed instruction is applied
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(204) @ 0: reporter [asm_gen] sub program name: sub
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1215) @ 0: reporter [asm_gen] Insert directed instr stream riscv_jal_instr 15/777 times
# ** Fatal: Cannot match types for literal init. to type 'enum int riscv_instr_pkg::riscv_instr_name_t' from type 'enum int riscv_instr_pkg::riscv_instr_name_t$[]'.
#    Time: 0 ns  Iteration: 61  Process: /uvm_pkg::uvm_task_phase::execute/#FORK#143(#ublk#215181159#143)_7feff05de77 File: /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv
# Fatal error in Function riscv_instr_pkg/riscv_jal_instr::post_randomize at /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv line 226
# 
# HDL call sequence:
# Stopped at /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv 226 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 1227 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 211 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 79 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/test/riscv_instr_base_test.sv 100 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_common_phases.svh 269 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_task_phase.svh 152 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# 
#  quit -f
# End time: 10:53:24 on Dec 21,2019, Elapsed time: 0:00:03
# Errors: 2, Warnings: 0
# *** Summary *********************************************
#     qrun: Errors:   0, Warnings:   0
#     vsim: Errors:   2, Warnings:   0
#   Totals: Errors:   2, Warnings:   0

Do you know reason why? Thank you

taoliug commented 4 years ago

I cannot reproduce this issue, might be something specific to qrun.

danghai commented 4 years ago

No, it still fail when I run with questa. Command: python3 run.py --test riscv_rand_instr_test -v --simulator questa -o questa

# ** Fatal: Cannot match types for literal init. to type 'enum int riscv_instr_pkg::riscv_instr_name_t' from type 'enum int riscv_instr_pkg::riscv_instr_name_t$[]'.
#    Time: 0 ns  Iteration: 61  Process: /uvm_pkg::uvm_task_phase::execute/#FORK#143(#ublk#215181159#143)_7feff1a5e3c File: /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv
# Fatal error in Function riscv_instr_pkg/riscv_jal_instr::post_randomize at /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv line 239
# 
# HDL call sequence:
# Stopped at /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv 239 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 1227 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 211 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 79 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/test/riscv_instr_base_test.sv 100 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_common_phases.svh 269 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_task_phase.svh 152 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# 
Fatal error reported during simulation. Coverage save is disabled. Please look above output for the fatal error message(s).
# End time: 23:24:08 on Dec 21,2019, Elapsed time: 0:00:04
# Errors: 2, Warnings: 0

It works for me before for both questa, and qrun when I run the full test cases.

danghai commented 4 years ago

It complains about Cannot match types for literal init. to type 'enum int riscv_instr_pkg::riscv_instr_name_t' from type 'enum int riscv_instr_pkg::riscv_instr_name_t$[]'. at riscv-dv/src/riscv_directed_instr_lib.sv line 239. It works for this test case: riscv_arithmetic_basic_test

taoliug commented 4 years ago

I tried to rewrite the code in a different way, please let me know if still see this issue.

danghai commented 4 years ago

I still got the same error. The log file:

# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1215) @ 0: reporter [asm_gen] Insert directed instr stream riscv_hazard_instr_stream 3/995 times
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1215) @ 0: reporter [asm_gen] Insert directed instr stream riscv_jal_instr 3/995 times
# ** Fatal: Cannot match types for literal init. to type 'enum int riscv_instr_pkg::riscv_instr_name_t' from type 'enum int riscv_instr_pkg::riscv_instr_name_t$[]'.
#    Time: 0 ns  Iteration: 61  Process: /uvm_pkg::uvm_task_phase::execute/#FORK#143(#ublk#215181159#143)_7feff05e3bc File: /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv
# Fatal error in Function riscv_instr_pkg/riscv_jal_instr::post_randomize at /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv line 239
# 
# HDL call sequence:
# Stopped at /home/danghai/hhoangda/riscv-dv/src/riscv_directed_instr_lib.sv 239 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 1227 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 211 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv 79 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/hhoangda/riscv-dv/test/riscv_instr_base_test.sv 100 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_common_phases.svh 269 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
# called from  /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_task_phase.svh 152 Function riscv_instr_pkg/riscv_jal_instr::post_randomize
#

danghai commented 4 years ago

It works before this commit HEAD is now at bb2b5a3 Migrate to new instruction class (#350) After you submited #350, it has error for both questa, and qrun

taoliug commented 4 years ago

It works for other simulators though. Let me check if i can implement in other way.

taoliug commented 4 years ago

Please sync and retry.

danghai commented 4 years ago

It still has the error. Now, the line of error is 216 at /riscv-dv/src/isa/riscv_instr.sv

# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_instr_sequence.sv(77) @ 0: reporter@@sub_1 [sub_1] Start generating 195 instruction
# ** Fatal: Cannot match types for literal init. to type 'enum int riscv_instr_pkg::riscv_instr_name_t' from type 'enum int riscv_instr_pkg::riscv_instr_name_t$[]'.
#    Time: 0 ns  Iteration: 61  Process: /uvm_pkg::uvm_task_phase::execute/#FORK#143(#ublk#215181159#143)_7feff05e3b7 File: /home/danghai/hhoangda/riscv-dv/src/isa/riscv_instr.sv
# Fatal error in Function riscv_instr_pkg/riscv_instr::get_rand_instr at /home/danghai/hhoangda/riscv-dv/src/isa/riscv_instr.sv line 216

taoliug commented 4 years ago

can you try again?

danghai commented 4 years ago

Yes. It works! Thank you.

taoliug commented 4 years ago

Cool, thanks for the confirmation

danghai commented 4 years ago

@taoliug Oh no, when I run the full test cases. It passes some test cases, and then fail at here

# /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): randomize() failed due to conflicts between the following constraints:
#   /home/danghai/hhoangda/riscv-dv/src/isa/riscv_compressed_instr.sv(9): rvc_csr_c { (rs1 inside { [S0:A5] }); }
#   /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (!(rs1 inside { local::this.cfg.reserved_regs }));
#   /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (!(rs1 == SP));
#   /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (rs1 inside { local::this.avail_regs });
# Where:
#   local::this.avail_regs = { A4,RA,T4,S10,T5,T0 }
#   local::this.cfg.reserved_regs = { S1,S0,A7 }
# Given:
#   bit [4:0] rs1
# ** Note: (vsim-7130) Enabling enhanced debug (-solvefaildebug=2) may generate a more descriptive constraint contradiction report.
#    Time: 0 ns  Iteration: 61  Region: /uvm_pkg::uvm_task_phase::execute
# ** Note: (vsim-7106) Use vsim option '-solvefailtestcase[=filename]' to generate a simplified testcase that will reproduce the failure.
#    Time: 0 ns  Iteration: 61  Region: /uvm_pkg::uvm_task_phase::execute
# ** Warning: (vsim-7084) No solutions exist which satisfy the specified constraints; randomize() failed.
#    Time: 0 ns  Iteration: 61  Process: /uvm_pkg::uvm_task_phase::execute/#FORK#143(#ublk#215181159#143)_7feff05e3b7 File: /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv Line: 233
# UVM_FATAL /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233) @ 0: reporter [riscv_hazard_instr_stream_4] Check failed (instr.randomize()) Randomization failed! 
# UVM_INFO verilog_src/uvm-1.2/src/base/uvm_report_server.svh(847) @ 0: reporter [UVM/REPORT/SERVER] 
# --- UVM Report Summary ---

taoliug commented 4 years ago

Can you paste a few more log before this failure? BTW, which test has this failure?

danghai commented 4 years ago

I am not sure which test makes error because the logfile is really long. I am watching, and the test start failing at

#    Time: 0 ns  Iteration: 269  Instance: /riscv_instr_gen_tb_top
# End time: 09:29:20 on Dec 23,2019, Elapsed time: 0:00:02
# Errors: 0, Warnings: 0
# *** Summary *********************************************
#     qrun: Errors:   0, Warnings:   0
#     vsim: Errors:   0, Warnings:   0
#   Totals: Errors:   0, Warnings:   0

Mon, 23 Dec 2019 09:29:20 run.py:209   INFO     Generating 2 riscv_hint_instr_test
Mon, 23 Dec 2019 09:29:20 run.py:219   INFO     Running riscv_hint_instr_test with 1 batches
Mon, 23 Dec 2019 09:29:20 run.py:248   INFO     Running riscv_hint_instr_test, batch 1/1, test_cnt:2
Mon, 23 Dec 2019 09:29:20 lib.py:108   DEBUG     qrun -64 -simulate -snapshot design_opt -c   -sv_seed 1720824916 -outdir /home/danghai/hhoangda/riscv-dv/qruntest/qrun.out +UVM_TESTNAME=riscv_rand_instr_test  +num_of_tests=2  +start_idx=0  +asm_file_name=qruntest/asm_tests/riscv_hint_instr_test  -l qruntest/sim_riscv_hint_instr_test_0.log +UVM_VERBOSITY=UVM_HIGH +hint_instr_ratio=5

taoliug commented 4 years ago

A few lines before this error will be helpful

/home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): randomize() failed due to conflicts between the following constraints:

danghai commented 4 years ago

Okay. The log file:

# -----------------------------------------------------------------------------
# Name                               Type                    Size  Value       
# -----------------------------------------------------------------------------
# cfg                                riscv_instr_gen_config  -     @794        
#   main_program_instr_cnt           integral                32    'h324       
#   sub_program_instr_cnt            sa(integral)            5     -           
#     [0]                            integral                32    'h662       
#     [1]                            integral                32    'h4ad       
#     [2]                            integral                32    'h2d7       
#     [3]                            integral                32    'hd3e       
#     [4]                            integral                32    'h8c8       
#   debug_program_instr_cnt          integral                32    'hd50b4df0  
#   data_page_pattern                data_pattern_t          2     INCR_VAL    
#   init_privileged_mode             privileged_mode_t       2     MACHINE_MODE
#   reserved_regs                    array(riscv_reg_t)      3     -           
#     [0]                            riscv_reg_t             5     S4          
#     [1]                            riscv_reg_t             5     S8          
#     [2]                            riscv_reg_t             5     T1          
#   ra                               riscv_reg_t             5     T1          
#   sp                               riscv_reg_t             5     S8          
#   tp                               riscv_reg_t             5     S4          
#   no_data_page                     integral                1     'h0         
#   no_branch_jump                   integral                1     'h0         
#   no_load_store                    integral                1     'h0         
#   no_csr_instr                     integral                1     'h0         
#   no_ebreak                        integral                1     'h1         
#   no_dret                          integral                1     'h1         
#   no_fence                         integral                1     'h0         
#   no_wfi                           integral                1     'h1         
#   enable_unaligned_load_store      integral                1     'h0         
#   illegal_instr_ratio              integral                32    'h0         
#   hint_instr_ratio                 integral                32    'h5         
#   boot_mode_opts                   string                  0     ""          
#   enable_page_table_exception      integral                32    'h0         
#   no_directed_instr                integral                1     'h0         
#   enable_interrupt                 integral                1     'h0         
#   enable_timer_irq                 integral                1     'h0         
#   bare_program_mode                integral                1     'h0         
#   enable_illegal_csr_instruction   integral                1     'h0         
#   enable_access_invalid_csr_level  integral                1     'h0         
#   enable_misaligned_instr          integral                1     'h0         
#   enable_dummy_csr_write           integral                1     'h0         
#   randomize_csr                    integral                1     'h0         
#   allow_sfence_exception           integral                1     'h0         
#   no_delegation                    integral                1     'h1         
#   force_m_delegation               integral                1     'h0         
#   force_s_delegation               integral                1     'h0         
#   support_supervisor_mode          integral                1     'h0         
#   disable_compressed_instr         integral                1     'h0         
#   signature_addr                   integral                32    'hdeadbeef  
#   require_signature_addr           integral                1     'h0         
#   gen_debug_section                integral                1     'h0         
#   enable_ebreak_in_debug_rom       integral                1     'h0         
#   set_dcsr_ebreak                  integral                1     'h0         
#   num_debug_sub_program            integral                32    'h0         
#   enable_debug_single_step         integral                1     'h0         
#   single_step_iterations           integral                32    'h643a9997  
#   set_mstatus_tw                   integral                1     'h0         
#   max_branch_step                  integral                32    'h14        
#   max_directed_instr_stream_seq    integral                32    'h14        
#   enable_floating_point            integral                1     'h0         
# -----------------------------------------------------------------------------
# 
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_load_store_rand_instr_stream ratio:4/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_loop_instr ratio:3/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_jal_instr ratio:4/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_hazard_instr_stream ratio:4/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_load_store_hazard_instr_stream ratio:4/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_multi_page_load_store_instr_stream ratio:4/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1161) @ 0: reporter [asm_gen] Adding directed instruction stream:riscv_mem_region_stress_test ratio:4/1000
# UVM_INFO /home/danghai/hhoangda/riscv-dv/test/riscv_instr_base_test.sv(99) @ 0: uvm_test_top [uvm_test_top] All directed instruction is applied
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(204) @ 0: reporter [asm_gen] sub program name: sub
# UVM_INFO /home/danghai/hhoangda/riscv-dv/src/riscv_asm_program_gen.sv(1215) @ 0: reporter [asm_gen] Insert directed instr stream riscv_hazard_instr_stream 6/1634 times
# /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): randomize() failed due to conflicts between the following constraints:
#   /home/danghai/hhoangda/riscv-dv/src/isa/riscv_compressed_instr.sv(9): rvc_csr_c { (rs1 inside { [S0:A5] }); }
#   /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (!(rs1 inside { local::this.cfg.reserved_regs }));
#   /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (!(rs1 == S1));
#   /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (rs1 inside { local::this.avail_regs });
# Where:
#   local::this.avail_regs = { A2,T6,T0,ZERO,GP,T3 }
#   local::this.cfg.reserved_regs = { S4,S8,T1 }
# Given:
#   bit [4:0] rs1
# ** Note: (vsim-7130) Enabling enhanced debug (-solvefaildebug=2) may generate a more descriptive constraint contradiction report.
#    Time: 0 ns  Iteration: 61  Region: /uvm_pkg::uvm_task_phase::execute
# ** Note: (vsim-7106) Use vsim option '-solvefailtestcase[=filename]' to generate a simplified testcase that will reproduce the failure.
#    Time: 0 ns  Iteration: 61  Region: /uvm_pkg::uvm_task_phase::execute
# ** Warning: (vsim-7084) No solutions exist which satisfy the specified constraints; randomize() failed.
#    Time: 0 ns  Iteration: 61  Process: /uvm_pkg::uvm_task_phase::execute/#FORK#143(#ublk#215181159#143)_7feff05e3b7 File: /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv Line: 233
# UVM_FATAL /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233) @ 0: reporter [riscv_hazard_instr_stream_1] Check failed (instr.randomize()) Randomization failed! 
# UVM_INFO verilog_src/uvm-1.2/src/base/uvm_report_server.svh(847) @ 0: reporter [UVM/REPORT/SERVER] 
# --- UVM Report Summary ---
# 
# ** Report counts by severity
# UVM_INFO : 1758
# UVM_WARNING :    0
# UVM_ERROR :    0
# UVM_FATAL :    1
# ** Report counts by id
# [Questa UVM]     3
# [RNTST]     1
# [UVM/RELNOTES]     1
# [asm_gen]    80
# [callstack_gen]     7
# [cfg]     1
# [data_page_gen]     5
# [illegal_instr]    47
# [main_program]   538
# [privil_seq]     1
# [riscv_hazard_instr_stream_1]     2
# [riscv_hazard_instr_stream_2]     2
# [riscv_hazard_instr_stream_3]     3
# [riscv_hazard_instr_stream_5]     2
# [riscv_hazard_instr_stream_6]     1
# [riscv_instr]   212
# [riscv_load_store_rand_instr_stream_3]     5
# [riscv_load_store_rand_instr_stream_5]     3
# [riscv_loop_instr_0]     6
# [riscv_loop_instr_1]     6
# [riscv_loop_instr_10]     1
# [riscv_loop_instr_11]     1
# [riscv_loop_instr_2]     5
# [riscv_loop_instr_3]     2
# [riscv_loop_instr_4]     2
# [riscv_loop_instr_5]     1
# [riscv_loop_instr_6]     1
# [riscv_loop_instr_7]     1
# [riscv_loop_instr_8]     1
# [riscv_loop_instr_9]     1
# [sub_1]   139
# [sub_2]   170
# [sub_3]   146
# [sub_4]   239
# [sub_5]   117
# [uvm_test_top]     6
# 
# ** Note: $finish    : /home/danghai/questasim/linux_x86_64/../verilog_src/uvm-1.2/src/base/uvm_root.svh(135)
#    Time: 0 ns  Iteration: 61  Region: /uvm_pkg::uvm_task_phase::execute
# End time: 09:47:30 on Dec 23,2019, Elapsed time: 0:00:03
# Errors: 0, Warnings: 1
# *** Summary *********************************************
#     qrun: Errors:   0, Warnings:   0
#     vsim: Errors:   0, Warnings:   1
#   Totals: Errors:   0, Warnings:   1

Mon, 23 Dec 2019 09:47:30 run.py:350   INFO     Compiling qruntest1/asm_tests/riscv_arithmetic_basic_test_0.S
Mon, 23 Dec 2019 09:47:30 run.py:351   DEBUG    /opt/riscv/bin/riscv64-unknown-elf-gcc -static -mcmodel=medany              -fvisibility=hidden -nostdlib              -nostartfiles qruntest1/asm_tests/riscv_arithmetic_basic_test_0.S              -I/home/danghai/hhoangda/riscv-dv/user_extension              -T/home/danghai/hhoangda/riscv-dv/scripts/link.ld  -o qruntest1/asm_tests/riscv_arithmetic_basic_test_0.o  -march=rv32imc -mabi=ilp32
Mon, 23 Dec 2019 09:47:30 run.py:353   DEBUG    b''
Mon, 23 Dec 2019 09:47:30 run.py:355   INFO     Converting to qruntest1/asm_tests/riscv_arithmetic_basic_test_0.bin
Mon, 23 Dec 2019 09:47:30 run.py:358   DEBUG    b''
Mon, 23 Dec 2019 09:47:30 run.py:350   INFO     Compiling qruntest1/asm_tests/riscv_arithmetic_basic_test_1.S
Mon, 23 Dec 2019 09:47:30 run.py:351   DEBUG    /opt/riscv/bin/riscv64-unknown-elf-gcc -static -mcmodel=medany              -fvisibility=hidden -nostdlib              -nostartfiles qruntest1/asm_tests/riscv_arithmetic_basic_test_1.S              -I/home/danghai/hhoangda/riscv-dv/user_extension              -T/home/danghai/hhoangda/riscv-dv/scripts/link.ld  -o qruntest1/asm_tests/riscv_arithmetic_basic_test_1.o  -march=rv32imc -mabi=ilp32
Mon, 23 Dec 2019 09:47:30 run.py:353   DEBUG    b''
Mon, 23 Dec 2019 09:47:30 run.py:355   INFO     Converting to qruntest1/asm_tests/riscv_arithmetic_basic_test_1.bin
Mon, 23 Dec 2019 09:47:30 run.py:358   DEBUG    b''
Mon, 23 Dec 2019 09:47:30 run.py:350   INFO     Compiling qruntest1/asm_tests/riscv_rand_instr_test_0.S
Mon, 23 Dec 2019 09:47:30 run.py:351   DEBUG    /opt/riscv/bin/riscv64-unknown-elf-gcc -static -mcmodel=medany              -fvisibility=hidden -nostdlib              -nostartfiles qruntest1/asm_tests/riscv_rand_instr_test_0.S              -I/home/danghai/hhoangda/riscv-dv/user_extension              -T/home/danghai/hhoangda/riscv-dv/scripts/link.ld  -o qruntest1/asm_tests/riscv_rand_instr_test_0.o  -march=rv32imc -mabi=ilp32
Mon, 23 Dec 2019 09:47:30 run.py:353   DEBUG    b''
Mon, 23 Dec 2019 09:47:30 run.py:355   INFO     Converting to qruntest1/asm_tests/riscv_rand_instr_test_0.bin
Mon, 23 Dec 2019 09:47:30 run.py:358   DEBUG    b''
Mon, 23 Dec 2019 09:47:30 run.py:350   INFO     Compiling qruntest1/asm_tests/riscv_rand_instr_test_1.S
Mon, 23 Dec 2019 09:47:30 run.py:351   DEBUG    /opt/riscv/bin/riscv64-unknown-elf-gcc -static -mcmodel=medany              -fvisibility=hidden -nostdlib              -nostartfiles qruntest1/asm_tests/riscv_rand_instr_test_1.S              -I/home/danghai/hhoangda/riscv-dv/user_extension              -T/home/danghai/hhoangda/riscv-dv/scripts/link.ld  -o qruntest1/asm_tests/riscv_rand_instr_test_1.o  -march=rv32imc -mabi=ilp32
riscv64-unknown-elf-gcc: error: qruntest1/asm_tests/riscv_rand_instr_test_1.S: No such file or directory
riscv64-unknown-elf-gcc: fatal error: no input files

We should implement to report which test is pass/fail for full regression test cases

taoliug commented 4 years ago

/home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): randomize() failed due to conflicts between the following constraints: /home/danghai/hhoangda/riscv-dv/src/isa/riscv_compressed_instr.sv(9): rvc_csr_c { (rs1 inside { [S0:A5] }); } /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (!(rs1 inside { local::this.cfg.reserved_regs })); /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (!(rs1 == S1)); /home/danghai/hhoangda/riscv-dv/src/riscv_instr_stream.sv(233): (rs1 inside { local::this.avail_regs }); Where: local::this.avail_regs = { A2,T6,T0,ZERO,GP,T3 } local::this.cfg.reserved_regs = { S4,S8,T1 }

I actually don't know why the constraint failed, s1 == A2 seems to be a valid result based on all these constraints.

danghai commented 4 years ago

Oh I know the reason why. That is my fault. The reason is "--noclean" switch. I do not specify the "--noclean" switch, and then It will clean it up for next test. It will work when I run: python3 run.py --simulator qrun --noclean. Should I make "--clean" switch instead of "--noclean"? It will not clean by default.

taoliug commented 4 years ago

I doubt if it's anything to do with noclean, the test should pass with a fresh output directory. What will happen if you manually remove all output directory, and run "python3 run.py --simulator qrun"?

danghai commented 4 years ago

It wont work for python3 run.py --simulator qrun It will work for python3 run.py --simulator qrun --noclean

taoliug commented 4 years ago

If you start from a clean directory, it shouldn't have any difference to run with or without --noclean. Does qrun save anything else locally outside the output folder? If yes, can you remove all qrun generated file and rerun the test?

danghai commented 4 years ago

I am really confused about the result. I do some experiment. qrun and questa has same behaviors. python3 run.py --simulator qrun --> Do not work python3 run.py --simulator qrun --noclean --> do not work

python3 run.py --simulator qrun
python3 run.py --simulator qrun --noclean  (re-run)  --> Can run full test cases

I really do not know why.

danghai commented 4 years ago

For more information, when I am able to run the full test cases by re-run. I got 5 mismatch in result.

Test binary: output/asm_tests/riscv_rand_instr_test_1.o
spike : output/spike_sim/riscv_rand_instr_test.1.csv
ovpsim : output/ovpsim_sim/riscv_rand_instr_test.1.csv
Mismatch[1]:
spike[10962] : csrrw   t6, mscratch, t6 -> t6(0x00000007) addr:0xffffffff80010040
ovpsim[10962] : lw      ra,4(t6) -> ra(0x00000000) addr:0x0000000080009824
Mismatch[2]:
spike[10963] : add     t6, a4, zero -> t6(0x00006000) addr:0xffffffff80010044
ovpsim[10963] : slt     s0,tp,t3 -> s0(0x00000001) addr:0x0000000080009828
Mismatch[3]:
spike[10964] : addi    t6, t6, -124 -> t6(0x00005f84) addr:0xffffffff80010048
ovpsim[10964] : sltiu   s3,s4,239 -> s3(0x00000001) addr:0x000000008000982c
Mismatch[4]:
spike[10965] : csrrw   t6, mscratch, t6 -> t6(0xb428b66d) addr:0xffffffff80010040
ovpsim[10965] : addi    t6,t6,28 -> t6(0xb428b689) addr:0x0000000080009830
Mismatch[5]:
spike[10966] : add     t6, a4, zero -> t6(0x00006000) addr:0xffffffff80010044
ovpsim[10966] : mulhsu  s3,s2,a6 -> s3(0x00000000) addr:0x0000000080009832
Mismatch[196686]:
[326174] spike : add     t6, a4, zero -> t6(0x00006000) addr:0xffffffff80010044
181113 instructions left in trace spike
[FAILED]: 22052 matched, 196686 mismatch

Test binary: output/asm_tests/riscv_rand_jump_test_1.o
spike : output/spike_sim/riscv_rand_jump_test.1.csv
ovpsim : output/ovpsim_sim/riscv_rand_jump_test.1.csv
Mismatch[1]:
spike[2158] : csrrw   sp, mscratch, sp -> sp(0x00000000) addr:0xffffffff8000c040
ovpsim[2158] : mulhu   s1,a7,t5 -> s1(0x00000000) addr:0x0000000080006b44
Mismatch[2]:
spike[2159] : add     sp, a5, zero -> sp(0x80021e7c) addr:0xffffffff8000c044
ovpsim[2159] : auipc   t0,0x11 -> t0(0x80017b48) addr:0x0000000080006b48
Mismatch[3]:
spike[2160] : addi    sp, sp, -124 -> sp(0x80021e00) addr:0xffffffff8000c048
ovpsim[2160] : addi    t0,t0,-1721 -> t0(0x8001748f) addr:0x0000000080006b4c
Mismatch[4]:
spike[2161] : csrr    s3, mepc -> s3(0x80006b42) addr:0xffffffff8000c08a
ovpsim[2161] : lb      ra,-20(t0) -> ra(0x00000000) addr:0x0000000080006b50
Mismatch[5]:
spike[2164] : csrr    s3, mcause -> s3(0x00000007) addr:0xffffffff8000c08e
ovpsim[2164] : mv      ra,s4 -> ra(0x00001000) addr:0x0000000080006b5c
361661 instructions left in trace ovpsim
[FAILED]: 3071 matched, 484550 mismatch

Test binary: output/asm_tests/riscv_illegal_instr_test_1.o
spike : output/spike_sim/riscv_illegal_instr_test.1.csv
ovpsim : output/ovpsim_sim/riscv_illegal_instr_test.1.csv
Mismatch[1]:
spike[178] : csrrw   sp, mscratch, sp -> sp(0x80020e1c) addr:0xffffffff80010040
ovpsim[178] : csrr    s8,mepc -> s8(0x800001f4) addr:0x000000008001008a
Mismatch[2]:
spike[179] : add     sp, a5, zero -> sp(0x00000002) addr:0xffffffff80010044
ovpsim[179] : csrr    s8,mcause -> s8(0x00000002) addr:0x000000008001008e
Mismatch[3]:
spike[180] : addi    sp, sp, -124 -> sp(0xffffff86) addr:0xffffffff80010048
ovpsim[180] : li      t5,3 -> t5(0x00000003) addr:0x0000000080010092
Mismatch[4]:
spike[182] : add     sp, a5, zero -> sp(0x00000002) addr:0xffffffff80010044
ovpsim[182] : li      t5,8 -> t5(0x00000008) addr:0x0000000080010098
Mismatch[5]:
spike[184] : addi    sp, sp, -124 -> sp(0xffffff86) addr:0xffffffff80010048
ovpsim[184] : li      t5,9 -> t5(0x00000009) addr:0x000000008001009e
[FAILED]: 106 matched, 6121 mismatch

Test binary: output/asm_tests/riscv_ebreak_debug_mode_test_0.o
spike : output/spike_sim/riscv_ebreak_debug_mode_test.0.csv
ovpsim : output/ovpsim_sim/riscv_ebreak_debug_mode_test.0.csv
Mismatch[1]:
[68] spike : csrrw   sp, mscratch, sp -> sp(0x00000000) addr:0xffffffff80010040
10483 instructions left in trace spike
[FAILED]: 59 matched, 1 mismatch

Test binary: output/asm_tests/riscv_unaligned_load_store_test_0.o
spike : output/spike_sim/riscv_unaligned_load_store_test.0.csv
ovpsim : output/ovpsim_sim/riscv_unaligned_load_store_test.0.csv
Mismatch[1]:
spike[66] : csrrw   sp, mscratch, sp -> sp(0x00000000) addr:0xffffffff8000c040
ovpsim[66] : lh      s10,3(a4) -> s10(0x00000000) addr:0x00000000800000d8
Mismatch[2]:
spike[68] : add     sp, t5, zero -> sp(0x80020e7c) addr:0xffffffff8000c044
ovpsim[68] : auipc   s4,0x16 -> s4(0x800160e0) addr:0x00000000800000e0
Mismatch[3]:
spike[69] : addi    sp, sp, -124 -> sp(0x80020e00) addr:0xffffffff8000c048
ovpsim[69] : addi    s4,s4,-777 -> s4(0x80015dd7) addr:0x00000000800000e4
Mismatch[4]:
spike[73] : csrr    s2, mepc -> s2(0x800000d8) addr:0xffffffff8000c08a
ovpsim[73] : lhu     a0,-4(s7) -> a0(0x00000000) addr:0x00000000800000f4
Mismatch[5]:
spike[74] : csrr    s2, mcause -> s2(0x00000004) addr:0xffffffff8000c08e
ovpsim[74] : lb      a1,-755(s4) -> a1(0x00000000) addr:0x00000000800000f8
17764 instructions left in trace ovpsim
[FAILED]: 57 matched, 17783 mismatch

taoliug commented 4 years ago

I feel there's some qrun tool issue, I don't see this syndrome with other simulators.

danghai commented 4 years ago

Do you get any mismatchs between spike and ovpsim?

taoliug commented 4 years ago

No mismatch on my side.

danghai commented 4 years ago

The result that I got is not stable. Sometime I got 4 failure, or 3 failure, or even 6 failure.

danghai commented 4 years ago

@taoliug . Follow up this issue. I compare all options between success command, and failure command. They are match except for random number for "-sv_seed". This random number has much effect on simulation? Simulation log without warning/fatal --> -sv_seed 2288139953 Simulation log with fatal --> sv_seed 252428313

Could you take the look? Thank you

taoliug commented 4 years ago

For the failed case, can you use vsim option '-solvefailtestcase[=filename]' to generate a simplified testcase?

danghai commented 4 years ago

I give you the test case that I generate.

taoliug commented 4 years ago

I can run it with no issue ... i=995: svseed=1 ; 5bd28f98429713b5 ; 1 i=996: svseed=1 ; 9c44d8e4d315c320 ; 1 i=997: svseed=1 ; 75803ea7efa7adef ; 1 i=998: svseed=1 ; d06a72267745a552 ; 1 i=999: svseed=1 ; 38eaf1750c633eb9 ; 1

Seems like a qrun tool issue to me.

danghai commented 4 years ago

It is weird. I think the reason why it works when I re-run command is probably the different random seed value.

taoliug commented 4 years ago

Tried another workaround, please check if it fixes the problem.

danghai commented 4 years ago

@taoliug , it works now.

chipsalliance / riscv-dv

Questa/Qrun simulator does not work for first run for full regression tests. It works when re-run with --noclean switch #374