search

verilog-to-routing / vtr-verilog-to-routing

Verilog to Routing -- Open Source CAD Flow for FPGA Research
https://verilogtorouting.org
Other
1.01k stars 391 forks source link

Nightly golden QoR is out of date? #1023

Closed litghost closed 5 years ago

litghost commented 5 years ago

Expected Behaviour

./run_reg_test.pl vtr_reg_nightly

should run without error.

Current Behaviour

@ f98a99ffc7d63c92e4d87d6b024e649c80a4c681

scripts/parse_vtr_task.pl -l /big/keithrothman/qor/baseline/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly/task_list.txt -check_golden -calc_geomean
regression_tests/vtr_reg_nightly/vpr_reg_mcnc...[Fail]                                                                             
 k6_frac_N10_40nm.xml/diffeq.pre-vpr.blif/common device_grid_tiles: golden = 100 result = 121
[Fail]                                                                                                              
 k6_frac_N10_40nm.xml/frisc.pre-vpr.blif/common device_grid_tiles: golden = 256 result = 225
regression_tests/vtr_reg_nightly/vtr_reg_qor_chain...[Fail]                                                           
 k6_frac_N10_frac_chain_mem32K_40nm.xml/arm_core.v/common num_pre_packed_nets: golden = 17413 result = 18548
[Fail]                                                                                                                   
 k6_frac_N10_frac_chain_mem32K_40nm.xml/arm_core.v/common num_pre_packed_blocks: golden = 17195 result = 18330
[Fail]                                                                                                
 k6_frac_N10_frac_chain_mem32K_40nm.xml/boundtop.v/common min_chan_width: golden = 48 result = 38
[Fail]                                                                                                           
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common num_pre_packed_nets: golden = 32456 result = 38222
[Fail]                                                                                                                 
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common num_pre_packed_blocks: golden = 28457 result = 33863
[Fail]                                                                                                                  
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common num_post_packed_blocks: golden = 1947 result = 2464
[Fail]                                                                                                                
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common device_width: golden = 50 result = 57
[Fail]                                                                                                                
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common device_height: golden = 50 result = 57
[Fail]                                                                                                                    
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common device_grid_tiles: golden = 2500 result = 3249
[Fail]                                                                                                 
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common device_limiting_resources: golden = clb memory result = clb
[Fail]                                                                                           
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU8PEEng.v/common num_clb: golden = 1679 result = 2196
[Fail]                                                                        
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common max_vpr_mem: golden = 1523996 result = 1979600
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common num_pre_packed_nets: golden = 102344 result = 124854
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common num_pre_packed_blocks: golden = 90079 result = 111149
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common num_post_packed_nets: golden = 48626 result = 59091
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common num_post_packed_blocks: golden = 5855 result = 7906
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common device_width: golden = 92 result = 102
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common device_height: golden = 92 result = 102
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common device_grid_tiles: golden = 8464 result = 10404
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common device_limiting_resources: golden = memory result = clb
[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/LU32PEEng.v/common num_clb: golden = 5442 result = 7491
regression_tests/vtr_reg_nightly/vtr_reg_qor_chain_depop...[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/arm_core.v/common num_pre_packed_nets: golden = 17413 result = 18548
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/arm_core.v/common num_pre_packed_blocks: golden = 17195 result = 18330
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU8PEEng.v/common num_pre_packed_nets: golden = 32456 result = 38222
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU8PEEng.v/common num_pre_packed_blocks: golden = 28457 result = 33863
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU8PEEng.v/common device_limiting_resources: golden = memory result = clb
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU8PEEng.v/common num_clb: golden = 1500 result = 1836
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common max_vpr_mem: golden = 1653416 result = 2389044
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common num_pre_packed_nets: golden = 102344 result = 124854
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common num_pre_packed_blocks: golden = 90079 result = 111149
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common num_post_packed_nets: golden = 47449 result = 58382
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common num_post_packed_blocks: golden = 5286 result = 6658
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common device_limiting_resources: golden = memory result = clb
[Fail]
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common num_clb: golden = 4873 result = 6243
regression_tests/vtr_reg_nightly/vtr_reg_netlist_writer...[Fail]
 k6_frac_N10_frac_chain_mem32K_40nm.xml/boundtop.v/common min_chan_width: golden = 48 result = 38
regression_tests/vtr_reg_nightly/vtr_func_formal...[Pass]
regression_tests/vtr_reg_nightly/titan_small...[Pass]
regression_tests/vtr_reg_nightly/titan_other...[Fail]
 stratixiv_arch.timing.xml/MCML_stratixiv_arch_timing.blif/common num_DSP: golden = 9 result = 10
regression_tests/vtr_reg_nightly/vtr_bidir...[Pass]
regression_tests/vtr_reg_nightly/complex_switch...[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU8PEEng.v/common num_pre_packed_nets: golden = 30929 result = 39002
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU8PEEng.v/common num_pre_packed_blocks: golden = 30831 result = 38728
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU8PEEng.v/common num_post_packed_nets: golden = 28252 result = 36304
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU8PEEng.v/common num_post_packed_blocks: golden = 3805 result = 4769
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU8PEEng.v/common num_clb: 
golden = 3500 result = 4462
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU32PEEng.v/common max_vpr_mem: golden = 2230644 result = 2768456
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU32PEEng.v/common num_pre_packed_nets: golden = 104755 result = 136869
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU32PEEng.v/common num_pre_packed_blocks: golden = 104057 result = 135467
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU32PEEng.v/common num_post_packed_nets: golden = 96665 result = 128734
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU32PEEng.v/common num_post_packed_blocks: golden = 12398 result = 16275
[Fail]
 k4_N8_topology-0.85sL2-0.15gL4-on-cb-off-sb_22nm_22nm.xml/LU32PEEng.v/common num_clb: golden = 11853 result = 15728
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_bidir...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_complex_switch...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_titan...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_error_check...[Pass]

Test 'vtr_reg_nightly' had 49 qor test failures

Test 'vtr_reg_nightly' had 0 run failures

Test complete

Error: 49 tests failed!

Possible Solution

It is unclear if the QoR regressions are expected or not. If they are expected, I guess the golden QoR needs to be updated.

Steps to Reproduce

Context

Your Environment

VPR FPGA Placement and Routing. Version: 8.0.0-rc1+unkown Revision: v8.0.0-rc1-1080-gf98a99ffc-dirty Compiled: 2019-10-29T23:04:27 Compiler: Clang 7.1.0 on Linux-4.9.0-7-amd64 x86_64 Build Info: release IPO VTR_ASSERT_LEVEL=2

kmurray commented 5 years ago

Yep, I noticed this too. Should be fixed with 4a559a23a