search

verilog-to-routing / vtr-verilog-to-routing

Verilog to Routing -- Open Source CAD Flow for FPGA Research
https://verilogtorouting.org
Other
1.01k stars 391 forks source link

Night and weekly QoR updates? #1233

Closed litghost closed 2 years ago

litghost commented 4 years ago

Expected Behaviour

Nightly and weekly QoR should be update to date

Current Behaviour

Nightly:

Calculating QoR results... 
scripts/parse_vtr_task.pl -l /tmpfs/src/github/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly/task_list.txt -check_golden -calc_geomean  
regression_tests/vtr_reg_nightly/vpr_reg_mcnc...[Pass]
regression_tests/vtr_reg_nightly/vtr_reg_qor_chain...[Pass]
regression_tests/vtr_reg_nightly/vtr_reg_qor_chain_depop...[Fail] 
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/bgm.v/common max_vpr_mem: golden = 621408 result = 764076
[Fail] 
 k6_frac_N10_frac_chain_depop50_mem32K_40nm.xml/LU32PEEng.v/common max_vpr_mem: golden = 2043204 result = 2475080
regression_tests/vtr_reg_nightly/vtr_reg_netlist_writer...[Pass]
regression_tests/vtr_reg_nightly/vtr_func_formal...[Pass]
regression_tests/vtr_reg_nightly/titan_quick_qor...[Pass]
regression_tests/vtr_reg_nightly/titan_other...[Fail] 
 stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -71115.7 result = -39987.9
[Fail] 
 stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common setup_TNS: golden = -76853.8 result = -44929.2
[Fail] 
 stratixiv_arch.timing.xml/CH_DFSIN_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -113037 result = -87082.9
[Fail] 
 stratixiv_arch.timing.xml/CH_DFSIN_stratixiv_arch_timing.blif/common setup_TNS: golden = -151525 result = -116397
[Fail] 
 stratixiv_arch.timing.xml/fir_cascade_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -299468 result = -135502
[Fail] 
 stratixiv_arch.timing.xml/fir_cascade_stratixiv_arch_timing.blif/common setup_TNS: golden = -322426 result = -151222
[Fail] 
 stratixiv_arch.timing.xml/jacobi_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -155165 result = -123598
[Fail] 
 stratixiv_arch.timing.xml/jacobi_stratixiv_arch_timing.blif/common setup_TNS: golden = -188495 result = -143727
[Fail] 
 stratixiv_arch.timing.xml/leon2_stratixiv_arch_timing.blif/common setup_WNS: golden = -8.52362 result = -6.72228
[Fail] 
 stratixiv_arch.timing.xml/leon3mp_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -110901 result = -85130.7
[Fail] 
 stratixiv_arch.timing.xml/leon3mp_stratixiv_arch_timing.blif/common setup_TNS: golden = -122910 result = -94790.3
[Fail] 
 stratixiv_arch.timing.xml/MCML_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -179533 result = -92521.6
[Fail] 
 stratixiv_arch.timing.xml/MCML_stratixiv_arch_timing.blif/common setup_TNS: golden = -240799 result = -145895
[Fail] 
 stratixiv_arch.timing.xml/MMM_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -112098 result = -43098.2
[Fail] 
 stratixiv_arch.timing.xml/MMM_stratixiv_arch_timing.blif/common setup_TNS: golden = -125550 result = -55002.7
[Fail] 
 stratixiv_arch.timing.xml/radar20_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -68289.9 result = -38182.9
[Fail] 
 stratixiv_arch.timing.xml/radar20_stratixiv_arch_timing.blif/common setup_TNS: golden = -79829.2 result = -50226.6
[Fail] 
 stratixiv_arch.timing.xml/radar20_stratixiv_arch_timing.blif/common setup_WNS: golden = -6.10098 result = -4.84686
[Fail] 
 stratixiv_arch.timing.xml/Reed_Solomon_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -53834.9 result = -34394
[Fail] 
 stratixiv_arch.timing.xml/Reed_Solomon_stratixiv_arch_timing.blif/common setup_TNS: golden = -58540.5 result = -38395.6
[Fail] 
 stratixiv_arch.timing.xml/Reed_Solomon_stratixiv_arch_timing.blif/common num_LAB: golden = 1119 result = 1175
[Fail] 
 stratixiv_arch.timing.xml/stap_steering_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -78074.1 result = -25559.1
[Fail] 
 stratixiv_arch.timing.xml/stap_steering_stratixiv_arch_timing.blif/common setup_TNS: golden = -84825.7 result = -29716.1
[Fail] 
 stratixiv_arch.timing.xml/stap_steering_stratixiv_arch_timing.blif/common setup_WNS: golden = -6.42913 result = -4.90307
[Fail] 
 stratixiv_arch.timing.xml/sudoku_check_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -40652.3 result = -22260.4
[Fail] 
 stratixiv_arch.timing.xml/sudoku_check_stratixiv_arch_timing.blif/common setup_TNS: golden = -46248.4 result = -27436.8
[Fail] 
 stratixiv_arch.timing.xml/SURF_desc_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -90036.8 result = -68065
[Fail] 
 stratixiv_arch.timing.xml/SURF_desc_stratixiv_arch_timing.blif/common setup_TNS: golden = -101418 result = -78216.4
[Fail] 
 stratixiv_arch.timing.xml/ucsb_152_tap_fir_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -14201.6 result = -4832.45
[Fail] 
 stratixiv_arch.timing.xml/ucsb_152_tap_fir_stratixiv_arch_timing.blif/common setup_TNS: golden = -15454.7 result = -5308.91
[Fail] 
 stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -68081.1 result = -38571.8
[Fail] 
 stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common setup_TNS: golden = -73939.8 result = -42390.1
[Fail] 
 stratixiv_arch.timing.xml/picosoc_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -56782.9 result = -39958.6
[Fail] 
 stratixiv_arch.timing.xml/picosoc_stratixiv_arch_timing.blif/common setup_TNS: golden = -62381 result = -44457.2
[Fail] 
 stratixiv_arch.timing.xml/murax_stratixiv_arch_timing.blif/common placed_setup_TNS_est: golden = -7029.52 result = -3594.94
[Fail] 
 stratixiv_arch.timing.xml/murax_stratixiv_arch_timing.blif/common setup_TNS: golden = -7444.8 result = -3858.25
regression_tests/vtr_reg_nightly/vtr_bidir...[Fail] 
 k4_n4_v7_bidir.xml/ex5p.blif/common critical_path_delay: golden = 26.6853 result = 15.5813
[Fail] 
 k4_n4_v7_bidir.xml/ex5p.blif/common setup_TNS: golden = -1099.95 result = -704.169
[Fail] 
 k4_n4_v7_bidir.xml/ex5p.blif/common setup_WNS: golden = -26.6853 result = -15.5813
[Fail] 
 k4_n4_v7_bidir.xml/spla.blif/common critical_path_delay: golden = 44.4049 result = 23.9953
[Fail] 
 k4_n4_v7_bidir.xml/spla.blif/common setup_TNS: golden = -1306.7 result = -829.511
[Fail] 
 k4_n4_v7_bidir.xml/spla.blif/common setup_WNS: golden = -44.4049 result = -23.9953
regression_tests/vtr_reg_nightly/complex_switch...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_bidir...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_complex_switch...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_titan...[Pass]
regression_tests/vtr_reg_nightly/vpr_verify_rr_graph_error_check...[Pass]

Test 'vtr_reg_nightly' had 44 qor test failures

Weekly:

[Fail] 
 stratixiv_arch.timing.xml/LU_Network_stratixiv_arch_timing.blif/common placed_CPD_est: golden = 9.87556 result = 6.44311
[Fail] 
 stratixiv_arch.timing.xml/LU_Network_stratixiv_arch_timing.blif/common placed_setup_WNS_est: golden = -9.87556 result = -6.44311
[Fail] 
 stratixiv_arch.timing.xml/LU_Network_stratixiv_arch_timing.blif/common critical_path_delay: golden = 10.2176 result = 7.07608
[Fail] 
 stratixiv_arch.timing.xml/LU_Network_stratixiv_arch_timing.blif/common setup_WNS: golden = -10.2176 result = -7.07608

Test 'vtr_reg_weekly' had 6 qor test failures

Test 'vtr_reg_weekly' had 1 run failures

Possible Solution

Update nightly and weekly QoR. All changes seem to be improvements?

Steps to Reproduce

  1. See Kokoro continuous nightly and weekly

Context

Your Environment

kmurray commented 4 years ago

Yep, agreed. I'm working on closing out the nightly updates in #1209

litghost commented 4 years ago

Yep, agreed. I'm working on closing out the nightly updates in #1209

Great!