search

The-OpenROAD-Project / OpenROAD-flow-scripts

OpenROAD's scripts implementing an RTL-to-GDS Flow. Documentation at https://openroad-flow-scripts.readthedocs.io/en/latest/
https://theopenroadproject.org/
Other
350 stars 292 forks source link

Failed at detailed routing stage 5_2_TritonRoute.log #1191

Open vishallparmar25 opened 1 year ago

vishallparmar25 commented 1 year ago

Subject

[Stage]: Detail Router.

Describe the bug

ORFS flow failed at detail route stage

[INFO DRT-0036] Pad guide region query size = 0. [INFO DRT-0179] Init gr pin query. [INFO DRT-0185] Post process initialize RPin region query. [INFO DRT-0181] Start track assignment. [INFO DRT-0184] Done with 726268 vertical wires in 15 frboxes and 499275 horizontal wires in 15 frboxes. [INFO DRT-0186] Done with 55361 vertical wires in 15 frboxes and 81996 horizontal wires in 15 frboxes. [INFO DRT-0182] Complete track assignment. [INFO DRT-0267] cpu time = 00:03:26, elapsed time = 00:01:00, memory = 5321.84 (MB), peak = 6472.33 (MB) [INFO DRT-0187] Start routing data preparation. [INFO DRT-0267] cpu time = 00:00:01, elapsed time = 00:00:01, memory = 5321.84 (MB), peak = 6472.33 (MB) [INFO DRT-0194] Start detail routing. [INFO DRT-0195] Start 0th optimization iteration. Command terminated by signal 9 Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.

Expected Behavior

[INFO DRT-0198] Complete detail routing. Total wire length = 127688 um. Total wire length on LAYER M1 = 0 um.

Environment

ubuntu@ip-172-31-10-38:~/Opensource/OpenROAD-flow-scripts/tools/OpenROAD$ ./etc/Env.sh 
[WARNING] Your current OpenROAD version is outdated.
It is recommened to pull the latest changes.
If problem persists, file a github issue with the re-producible test case.
kernel: Linux 5.19.0-1027-aws
os: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
cmake version 3.24.2
-- The CXX compiler identification is GNU 11.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-8118-gb95ce9901
-- System name: Linux
-- Compiler: GNU 11.3.0
-- Build type: RELEASE
-- Install prefix: /usr/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- Found GTest: /usr/lib/x86_64-linux-gnu/cmake/GTest/GTestConfig.cmake (found version "1.11.0")  
-- The C compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- TCL readline library: /usr/lib/x86_64-linux-gnu/libtclreadline.so
-- TCL readline header: /usr/include/x86_64-linux-gnu
-- Found SWIG: /usr/local/bin/swig (found suitable version "4.1.0", minimum required is "3.0")  
-- Using SWIG >= 4.1.0 -flatstaticmethod flag for python
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0")  
-- boost: 1.80.0
-- Found Python3: /usr/include/python3.10 (found version "3.10.6") found components: Development Development.Module Development.Embed 
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
-- Found Threads: TRUE  
-- spdlog: 1.8.1
-- Found BISON: /usr/bin/bison (found version "3.8.2") 
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- STA version: 2.4.0
-- STA git sha: 555493cba6e476bd2ff0b9a543de7a781276c2b3
-- System name: Linux
-- Compiler: GNU 11.3.0
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /usr/bin/flex (found version "2.6.4") 
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- SSTA: 0
-- STA executable: /home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/OpenROAD/src/sta/app/sta
-- GPU is not enabled
-- Found re2: /opt/or-tools/lib/cmake/re2/re2Config.cmake (found version "9.0.0") 
-- Found Clp: /opt/or-tools/lib/cmake/Clp/ClpConfig.cmake (found version "1.17.7") 
-- Found Cbc: /opt/or-tools/lib/cmake/Cbc/CbcConfig.cmake (found version "2.10.7") 
-- Found Eigen3: /usr/local/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0") 
-- Found SCIP: /opt/or-tools/lib/cmake/scip/scip-config.cmake (found version "8.0.1") 
-- GUI is enabled
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Could NOT find VTune (missing: VTune_LIBRARIES VTune_INCLUDE_DIRS) 
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found suitable version "1.80.0", minimum required is "1.78")  
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization system thread 
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0")  
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp.cMNb5fVXYM

To Reproduce

export RCX_RULES='./platforms/asap7/rcx_patterns.rules' export DESIGN_NAME='tpu_top' export SDC_FILE='./designs/asap7/TPU_TOP/constraint.sdc' export IO_PLACER_H='M2 M4 M6 M8' export GDS_FILES='./platforms/asap7/gds/asap7sc7p5t_28_R_220121a.gds' export MAKE_TRACKS='./platforms/asap7/openRoad/make_tracks.tcl' export NPROC='4' export TAP_CELL_NAME='TAPCELL_ASAP7_75t_R' export HIER_REPORT_SCRIPT='./scripts/synth_hier_report.tcl' export RTLMP_FLOW='True' export CTS_BUF_CELL='BUFx4_ASAP7_75t_R' export STREAM_SYSTEM='GDS' export SYNTH_HIERARCHICAL='1' export LATCH_MAP_FILE='./platforms/asap7/yoSys/cells_latch_R.v' export TC_VOLTAGE='0.70' export WC_DFF_LIB_FILE='./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_SS_nldm_220123.lib' export TAPCELL_TCL='./platforms/asap7/openRoad/tapcell.tcl' export BC_VOLTAGE='0.77' export VOLTAGE='0.77' export TCLLIBPATH='util/cell-veneer' export GPL_ROUTABILITY_DRIVEN='1' export WC_TEMPERATURE='100C' export WC_LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_SS_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_SS_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_SS_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_SS_nldm_220123.lib ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_SS_nldm_211120.lib.gz' export OBJECTS_DIR='./objects/asap7/TPU_TOP/base' export DESIGN_NICKNAME='TPU_TOP' export ENABLE_DPO='1' export BC_TEMPERATURE='25C' export HAS_IO_CONSTRAINTS='1' export YOSYS_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys' export RTLMP_RPT_DIR='./objects/asap7/TPU_TOP/base/rtlmp' export SYNTH_SCRIPT='./scripts/synth.tcl' export ABC_DRIVER_CELL='BUFx2_ASAP7_75t_R' export PWR_NETS_VOLTAGES='"VDD 0.77"' export BC_LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export CURDIR='/home/ubuntu/Opensource/OpenROAD-flow-scripts/flow' export TIELO_CELL_AND_PORT='TIELOx1_ASAP7_75t_R L' export DESIGN_HOME='./designs' export SCRIPTS_DIR='./scripts' export IO_PLACER_V='M3 M5 M7 M9' export MAX_UNGROUP_SIZE='0' export RESYNTH_TIMING_RECOVER='0' export SYNTH_ARGS='-flatten' export ABC_AREA='1' export CTS_BUF_DISTANCE='60' export LSORACLE_PLUGIN='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/yosys/share/yosys/plugin/oracle.so' export LOG_DIR='./logs/asap7/TPU_TOP/base' export CELL_PAD_IN_SITES_DETAIL_PLACEMENT='2' export TECH_LEF='./platforms/asap7/lef/asap7_tech_1x_201209.lef' export GND_NETS_VOLTAGES='"VSS 0.0"' export DIE_AREA='0 0 400 400' export RESYNTH_AREA_RECOVER='0' export ISSUE_SCRIPTS='add_routing_blk cdl cts deleteNonClkNets deletePowerNets deleteRoutingObstructions density_fill detail_place detail_route fillcell final_report floorplan generate_abstract global_place global_place_skip_io global_route gui io_placement io_placement_random klayout load macro_place pdn placement_blockages read_liberty read_macro_placement report_metrics resize run_all save_images synth synth_hier_report synth_preamble tapcell tdms_place view_cells write_def write_ref_sdc write_verilog yosys' export PLATFORM='asap7' export LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export SET_RC_TCL='./platforms/asap7/setRC.tcl' export GALLERY_REPORT='0' export SHELL='/usr/bin/env bash' export FILL_CELLS='FILLERxp5_ASAP7_75t_R FILLER_ASAP7_75t_R DECAPx1_ASAP7_75t_R DECAPx2_ASAP7_75t_R DECAPx4_ASAP7_75t_R DECAPx6_ASAP7_75t_R DECAPx10_ASAP7_75t_R' export OPENROAD_NO_EXIT_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -no_init' export OPENROAD_EXE='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad' export TNS_END_PERCENT='5' export GPL_TIMING_DRIVEN='1' export CELL_PAD_IN_SITES_GLOBAL_PLACEMENT='4' export MIN_BUF_CELL_AND_PORTS='BUFx2_ASAP7_75t_R A Y' export PLACE_DENSITY='0.25' export RESULTS_DIR='./results/asap7/TPU_TOP/base' export STDBUF_CMD='stdbuf -o L' export CLKGATE_MAP_FILE='./platforms/asap7/yoSys/cells_clkgate_R.v' export REPORTS_DIR='./reports/asap7/TPU_TOP/base' export PLATFORM_DIR='./platforms/asap7' export PLATFORM_HOME='./platforms' export GDSOAS_FILES='./platforms/asap7/gds/asap7sc7p5t_28_R_220121a.gds' export LSORACLE_KAHYPAR_CONFIG='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/LSOracle/share/lsoracle/test.ini' export WORK_HOME='.' export TIEHI_CELL_AND_PORT='TIEHIx1_ASAP7_75t_R H' export DESIGN_CONFIG='./designs/asap7/TPU_TOP/config.mk' export DONT_USE_SC_LIB='./objects/asap7/TPU_TOP/base/lib/merged.lib' export BC_DFF_LIB_FILE='./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export CORE_AREA='0.05 0.05 399.95 399.95' export STREAM_SYSTEM_EXT='gds' export TC_TEMPERATURE='0C' export ADDER_MAP_FILE='./platforms/asap7/yoSys/cells_adders_R.v' export LSORACLE_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/LSOracle/bin/lsoracle' export PLACE_SITE='asap7sc7p5t' export YOSYS_FLAGS='-v 3' export SYNTH_STOP_MODULE_SCRIPT='./objects/asap7/TPU_TOP/base/mark_hier_stop_modules.tcl' export DESIGN_DIR='./designs/asap7/TPU_TOP/' export TEMPERATURE='25C' export TC_LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_TT_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_TT_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_TT_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_TT_nldm_220123.lib ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_TT_nldm_211120.lib.gz' export FLOW_HOME='.' export MACRO_PLACE_CHANNEL='12 12' export DONT_USE_CELLS='x1p_ASAP7 xp_ASAP7 SDF ICG DFFH' export SC_LEF='./platforms/asap7/lef/asap7sc7p5t_28_R_1x_220121a.lef' export RESULTS_ODB='2_1_floorplan.odb 2_2_floorplan_io.odb 2_3_floorplan_tdms.odb 2_4_floorplan_macro.odb 2_5_floorplan_tapcell.odb 2_6_floorplan_pdn.odb 2_floorplan.odb 3_1_place_gp_skip_io.odb 3_2_place_iop.odb 3_3_place_gp.odb 3_4_place_resized.odb 3_5_place_dp.odb 3_place.odb 4_1_cts.odb 4_2_cts_fillcell.odb 4_cts.odb 5_1_grt.odb' export TEST_DIR='./test' export MACRO_PLACE_HALO='10 10' export DONT_USE_LIBS=' ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export PUBLIC='nangate45 sky130hd sky130hs asap7' export WRAP_CFG='./platforms/asap7/wrapper.cfg' export DPO_MAX_DISPLACEMENT='5 1' export GDS_FINAL_FILE='./results/asap7/TPU_TOP/base/6_final.gds' export VERILOG_FILES='./designs/src/TPU_TOP/addr_sel.v ./designs/src/TPU_TOP/quantize.v ./designs/src/TPU_TOP/sram_256x32b.v ./designs/src/TPU_TOP/sram_64x512b.v ./designs/src/TPU_TOP/systolic.v ./designs/src/TPU_TOP/systolic_control.v ./designs/src/TPU_TOP/tpu_top.v ./designs/src/TPU_TOP/write_out.v' export PDN_TCL='./platforms/asap7/openRoad/pdn/grid_strategy-M1-M2-M5-M6.tcl' export TIME_TEST='foo' export UTILS_DIR='./util' export ABC_CLOCK_PERIOD_IN_PS='1000' export MIN_ROUTING_LAYER='M2' export PROCESS='7' export GDS_MERGED_FILE='./results/asap7/TPU_TOP/base/6_1_merged.gds' export TEST_SCRIPT='./test/core_tests.sh' export NUM_CORES='4' export OPENROAD_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -exit -no_init' export ABC_LOAD_IN_FF='3.898' export MAX_ROUTING_LAYER='M9' export CORNER='BC' export OPENROAD_ARGS='-no_init' export OPENROAD_GUI_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -gui' export WC_VOLTAGE='0.63' export PLACE_PINS_ARGS='-min_distance 20 -min_distance_in_tracks' export TC_DFF_LIB_FILE='./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_TT_nldm_220123.lib' export FLOW_VARIANT='base' export RULES_DESIGN='./designs/asap7/TPU_TOP/rules-base.json' export RTLMP_BLOCKAGE_FILE='./objects/asap7/TPU_TOP/base/rtlmp/partition.txt.blockage' export RTLMP_RPT_FILE='partition.txt' export IR_DROP_LAYER='M1' export GDS_ALLOW_EMPTY='fakeram.'

Relevant log output

OpenROAD v2.0-8474-gcb591f10a 
This program is licensed under the BSD-3 license. See the LICENSE file for details.
Components of this program may be licensed under more restrictive licenses which must be honored.
[INFO ORD-0030] Using 4 thread(s).
detailed_route arguments:  -bottom_routing_layer M2 -top_routing_layer M9 -save_guide_updates -verbose 1
[INFO DRT-0149] Reading tech and libs.
[WARNING DRT-0140] SpacingRange unsupported.
[WARNING DRT-0145] New SPACINGTABLE PARALLELRUNLENGTH overrides old SPACING rule.
[WARNING DRT-0145] New SPACINGTABLE PARALLELRUNLENGTH overrides old SPACING rule.
[WARNING DRT-0145] New SPACINGTABLE PARALLELRUNLENGTH overrides old SPACING rule.
[WARNING DRT-0145] New SPACINGTABLE PARALLELRUNLENGTH overrides old SPACING rule.

Units:                1000
Number of layers:     21
Number of macros:     212
Number of vias:       9
Number of viarulegen: 11

[INFO DRT-0150] Reading design.

Design:                   tpu_top
Die area:                 ( 0 0 ) ( 400000 400000 )
Number of track patterns: 32
Number of DEF vias:       0
Number of components:     866832
Number of terminals:      1129
Number of snets:          2
Number of nets:           207490

[WARNING DRT-0240] CUT layer V3 does not have square single-cut via, cut layer width may be set incorrectly.
[WARNING DRT-0240] CUT layer V5 does not have square single-cut via, cut layer width may be set incorrectly.
[INFO DRT-0167] List of default vias:
  Layer V1
    default via: VIA12
  Layer V2
    default via: VIA23
  Layer V3
    default via: VIA34
  Layer V4
    default via: VIA45
  Layer V5
    default via: VIA56
  Layer V6
    default via: VIA67
  Layer V7
    default via: VIA78
  Layer V8
    default via: VIA89
  Layer V9
    default via: VIA9Pad
[INFO DRT-0162] Library cell analysis.
[INFO DRT-0163] Instance analysis.
  Complete 10000 instances.
  Complete 20000 instances.
  Complete 30000 instances.
  Complete 40000 instances.
  Complete 50000 instances.
  Complete 60000 instances.
  Complete 70000 instances.
  Complete 80000 instances.
  Complete 90000 instances.
  Complete 100000 instances.
  Complete 200000 instances.
  Complete 300000 instances.
  Complete 400000 instances.
  Complete 500000 instances.
  Complete 600000 instances.
  Complete 700000 instances.
  Complete 800000 instances.
[INFO DRT-0164] Number of unique instances = 318.
[INFO DRT-0168] Init region query.
[INFO DRT-0018]   Complete 10000 insts.
[INFO DRT-0018]   Complete 20000 insts.
[INFO DRT-0018]   Complete 30000 insts.
[INFO DRT-0018]   Complete 40000 insts.
[INFO DRT-0018]   Complete 50000 insts.
[INFO DRT-0018]   Complete 60000 insts.
[INFO DRT-0018]   Complete 70000 insts.
[INFO DRT-0018]   Complete 80000 insts.
[INFO DRT-0018]   Complete 90000 insts.
[INFO DRT-0019]   Complete 100000 insts.
[INFO DRT-0019]   Complete 200000 insts.
[INFO DRT-0019]   Complete 300000 insts.
[INFO DRT-0019]   Complete 400000 insts.
[INFO DRT-0019]   Complete 500000 insts.
[INFO DRT-0019]   Complete 600000 insts.
[INFO DRT-0019]   Complete 700000 insts.
[INFO DRT-0019]   Complete 800000 insts.
[INFO DRT-0024]   Complete Active.
[INFO DRT-0024]   Complete V0.
[INFO DRT-0024]   Complete M1.
[INFO DRT-0024]   Complete V1.
[INFO DRT-0024]   Complete M2.
[INFO DRT-0024]   Complete V2.
[INFO DRT-0024]   Complete M3.
[INFO DRT-0024]   Complete V3.
[INFO DRT-0024]   Complete M4.
[INFO DRT-0024]   Complete V4.
[INFO DRT-0024]   Complete M5.
[INFO DRT-0024]   Complete V5.
[INFO DRT-0024]   Complete M6.
[INFO DRT-0024]   Complete V6.
[INFO DRT-0024]   Complete M7.
[INFO DRT-0024]   Complete V7.
[INFO DRT-0024]   Complete M8.
[INFO DRT-0024]   Complete V8.
[INFO DRT-0024]   Complete M9.
[INFO DRT-0024]   Complete V9.
[INFO DRT-0024]   Complete Pad.
[INFO DRT-0033] Active shape region query size = 0.
[INFO DRT-0033] V0 shape region query size = 0.
[INFO DRT-0033] M1 shape region query size = 6594886.
[INFO DRT-0033] V1 shape region query size = 16751950.
[INFO DRT-0033] M2 shape region query size = 263939.
[INFO DRT-0033] V2 shape region query size = 151062.
[INFO DRT-0033] M3 shape region query size = 302524.
[INFO DRT-0033] V3 shape region query size = 100708.
[INFO DRT-0033] M4 shape region query size = 251770.
[INFO DRT-0033] V4 shape region query size = 100708.
[INFO DRT-0033] M5 shape region query size = 103088.
[INFO DRT-0033] V5 shape region query size = 4624.
[INFO DRT-0033] M6 shape region query size = 2380.
[INFO DRT-0033] V6 shape region query size = 0.
[INFO DRT-0033] M7 shape region query size = 0.
[INFO DRT-0033] V7 shape region query size = 0.
[INFO DRT-0033] M8 shape region query size = 200.
[INFO DRT-0033] V8 shape region query size = 0.
[INFO DRT-0033] M9 shape region query size = 379.
[INFO DRT-0033] V9 shape region query size = 0.
[INFO DRT-0033] Pad shape region query size = 0.
[INFO DRT-0165] Start pin access.
[INFO DRT-0076]   Complete 100 pins.
[INFO DRT-0076]   Complete 200 pins.
[INFO DRT-0076]   Complete 300 pins.
[INFO DRT-0076]   Complete 400 pins.
[INFO DRT-0076]   Complete 500 pins.
[INFO DRT-0076]   Complete 600 pins.
[INFO DRT-0076]   Complete 700 pins.
[INFO DRT-0076]   Complete 800 pins.
[INFO DRT-0078]   Complete 889 pins.
[INFO DRT-0079]   Complete 100 unique inst patterns.
[INFO DRT-0079]   Complete 200 unique inst patterns.
[INFO DRT-0081]   Complete 284 unique inst patterns.
[INFO DRT-0082]   Complete 1000 groups.
[INFO DRT-0082]   Complete 2000 groups.
[INFO DRT-0082]   Complete 3000 groups.
[INFO DRT-0082]   Complete 4000 groups.
[INFO DRT-0082]   Complete 5000 groups.
[INFO DRT-0082]   Complete 6000 groups.
[INFO DRT-0082]   Complete 7000 groups.
[INFO DRT-0082]   Complete 8000 groups.
[INFO DRT-0082]   Complete 9000 groups.
[INFO DRT-0083]   Complete 10000 groups.
[INFO DRT-0083]   Complete 20000 groups.
[INFO DRT-0083]   Complete 30000 groups.
[INFO DRT-0083]   Complete 40000 groups.
[INFO DRT-0083]   Complete 50000 groups.
[INFO DRT-0083]   Complete 60000 groups.
[INFO DRT-0083]   Complete 70000 groups.
[INFO DRT-0083]   Complete 80000 groups.
[INFO DRT-0083]   Complete 90000 groups.
[INFO DRT-0083]   Complete 100000 groups.
[INFO DRT-0083]   Complete 110000 groups.
[INFO DRT-0083]   Complete 120000 groups.
[INFO DRT-0083]   Complete 130000 groups.
[INFO DRT-0083]   Complete 140000 groups.
[INFO DRT-0083]   Complete 150000 groups.
[INFO DRT-0083]   Complete 160000 groups.
[INFO DRT-0083]   Complete 170000 groups.
[INFO DRT-0084]   Complete 177221 groups.
#scanned instances     = 866832
#unique  instances     = 302
#stdCellGenAp          = 11605
#stdCellValidPlanarAp  = 120
#stdCellValidViaAp     = 9347
#stdCellPinNoAp        = 0
#stdCellPinCnt         = 560151
#instTermValidViaApCnt = 0
#macroGenAp            = 0
#macroValidPlanarAp    = 0
#macroValidViaAp       = 0
#macroNoAp             = 0
[INFO DRT-0166] Complete pin access.
[INFO DRT-0267] cpu time = 00:01:05, elapsed time = 00:00:18, memory = 2617.67 (MB), peak = 3128.44 (MB)
[INFO DRT-0156] guideIn read 100000 guides.
[INFO DRT-0156] guideIn read 200000 guides.
[INFO DRT-0156] guideIn read 300000 guides.
[INFO DRT-0156] guideIn read 400000 guides.
[INFO DRT-0156] guideIn read 500000 guides.
[INFO DRT-0156] guideIn read 600000 guides.
[INFO DRT-0156] guideIn read 700000 guides.
[INFO DRT-0156] guideIn read 800000 guides.
[INFO DRT-0156] guideIn read 900000 guides.
[INFO DRT-0157] guideIn read 1000000 guides.

Number of guides:     1498450

[INFO DRT-0169] Post process guides.
[INFO DRT-0176] GCELLGRID X 0 DO 740 STEP 540 ;
[INFO DRT-0177] GCELLGRID Y 0 DO 740 STEP 540 ;
[INFO DRT-0026]   Complete 10000 origin guides.
[INFO DRT-0026]   Complete 20000 origin guides.
[INFO DRT-0026]   Complete 30000 origin guides.
[INFO DRT-0026]   Complete 40000 origin guides.
[INFO DRT-0026]   Complete 50000 origin guides.
[INFO DRT-0026]   Complete 60000 origin guides.
[INFO DRT-0026]   Complete 70000 origin guides.
[INFO DRT-0026]   Complete 80000 origin guides.
[INFO DRT-0026]   Complete 90000 origin guides.
[INFO DRT-0027]   Complete 100000 origin guides.
[INFO DRT-0027]   Complete 200000 origin guides.
[INFO DRT-0027]   Complete 300000 origin guides.
[INFO DRT-0027]   Complete 400000 origin guides.
[INFO DRT-0027]   Complete 500000 origin guides.
[INFO DRT-0027]   Complete 600000 origin guides.
[INFO DRT-0027]   Complete 700000 origin guides.
[INFO DRT-0027]   Complete 800000 origin guides.
[INFO DRT-0027]   Complete 900000 origin guides.
[INFO DRT-0027]   Complete 1000000 origin guides.
[INFO DRT-0027]   Complete 1100000 origin guides.
[INFO DRT-0027]   Complete 1200000 origin guides.
[INFO DRT-0027]   Complete 1300000 origin guides.
[INFO DRT-0027]   Complete 1400000 origin guides.
[INFO DRT-0028]   Complete Active.
[INFO DRT-0028]   Complete V0.
[INFO DRT-0028]   Complete M1.
[INFO DRT-0028]   Complete V1.
[INFO DRT-0028]   Complete M2.
[INFO DRT-0028]   Complete V2.
[INFO DRT-0028]   Complete M3.
[INFO DRT-0028]   Complete V3.
[INFO DRT-0028]   Complete M4.
[INFO DRT-0028]   Complete V4.
[INFO DRT-0028]   Complete M5.
[INFO DRT-0028]   Complete V5.
[INFO DRT-0028]   Complete M6.
[INFO DRT-0028]   Complete V6.
[INFO DRT-0028]   Complete M7.
[INFO DRT-0028]   Complete V7.
[INFO DRT-0028]   Complete M8.
[INFO DRT-0028]   Complete V8.
[INFO DRT-0028]   Complete M9.
[INFO DRT-0028]   Complete V9.
[INFO DRT-0028]   Complete Pad.
  complete 10000 nets.
  complete 20000 nets.
  complete 30000 nets.
  complete 40000 nets.
  complete 50000 nets.
  complete 60000 nets.
  complete 70000 nets.
  complete 80000 nets.
  complete 90000 nets.
  complete 100000 nets.
  complete 200000 nets.
[INFO DRT-0178] Init guide query.
[INFO DRT-0029]   Complete 10000 nets (guide).
[INFO DRT-0029]   Complete 20000 nets (guide).
[INFO DRT-0029]   Complete 30000 nets (guide).
[INFO DRT-0029]   Complete 40000 nets (guide).
[INFO DRT-0029]   Complete 50000 nets (guide).
[INFO DRT-0029]   Complete 60000 nets (guide).
[INFO DRT-0029]   Complete 70000 nets (guide).
[INFO DRT-0029]   Complete 80000 nets (guide).
[INFO DRT-0029]   Complete 90000 nets (guide).
[INFO DRT-0030]   Complete 100000 nets (guide).
[INFO DRT-0030]   Complete 200000 nets (guide).
[INFO DRT-0035]   Complete Active (guide).
[INFO DRT-0035]   Complete V0 (guide).
[INFO DRT-0035]   Complete M1 (guide).
[INFO DRT-0035]   Complete V1 (guide).
[INFO DRT-0035]   Complete M2 (guide).
[INFO DRT-0035]   Complete V2 (guide).
[INFO DRT-0035]   Complete M3 (guide).
[INFO DRT-0035]   Complete V3 (guide).
[INFO DRT-0035]   Complete M4 (guide).
[INFO DRT-0035]   Complete V4 (guide).
[INFO DRT-0035]   Complete M5 (guide).
[INFO DRT-0035]   Complete V5 (guide).
[INFO DRT-0035]   Complete M6 (guide).
[INFO DRT-0035]   Complete V6 (guide).
[INFO DRT-0035]   Complete M7 (guide).
[INFO DRT-0035]   Complete V7 (guide).
[INFO DRT-0035]   Complete M8 (guide).
[INFO DRT-0035]   Complete V8 (guide).
[INFO DRT-0035]   Complete M9 (guide).
[INFO DRT-0035]   Complete V9 (guide).
[INFO DRT-0035]   Complete Pad (guide).
[INFO DRT-0036] Active guide region query size = 0.
[INFO DRT-0036] V0 guide region query size = 0.
[INFO DRT-0036] M1 guide region query size = 457630.
[INFO DRT-0036] V1 guide region query size = 0.
[INFO DRT-0036] M2 guide region query size = 476669.
[INFO DRT-0036] V2 guide region query size = 0.
[INFO DRT-0036] M3 guide region query size = 263071.
[INFO DRT-0036] V3 guide region query size = 0.
[INFO DRT-0036] M4 guide region query size = 20346.
[INFO DRT-0036] V4 guide region query size = 0.
[INFO DRT-0036] M5 guide region query size = 4453.
[INFO DRT-0036] V5 guide region query size = 0.
[INFO DRT-0036] M6 guide region query size = 1618.
[INFO DRT-0036] V6 guide region query size = 0.
[INFO DRT-0036] M7 guide region query size = 735.
[INFO DRT-0036] V7 guide region query size = 0.
[INFO DRT-0036] M8 guide region query size = 642.
[INFO DRT-0036] V8 guide region query size = 0.
[INFO DRT-0036] M9 guide region query size = 379.
[INFO DRT-0036] V9 guide region query size = 0.
[INFO DRT-0036] Pad guide region query size = 0.
[INFO DRT-0179] Init gr pin query.
[INFO DRT-0185] Post process initialize RPin region query.
[INFO DRT-0181] Start track assignment.
[INFO DRT-0184] Done with 726268 vertical wires in 15 frboxes and 499275 horizontal wires in 15 frboxes.
[INFO DRT-0186] Done with 55361 vertical wires in 15 frboxes and 81996 horizontal wires in 15 frboxes.
[INFO DRT-0182] Complete track assignment.
[INFO DRT-0267] cpu time = 00:03:26, elapsed time = 00:01:00, memory = 5321.84 (MB), peak = 6472.33 (MB)
[INFO DRT-0187] Start routing data preparation.
[INFO DRT-0267] cpu time = 00:00:01, elapsed time = 00:00:01, memory = 5321.84 (MB), peak = 6472.33 (MB)
[INFO DRT-0194] Start detail routing.
[INFO DRT-0195] Start 0th optimization iteration.
Command terminated by signal 9
Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.

Screenshots

config.mk

image

Instance Count image

Placement Density heat map image

Routing congestion heat map image

Additional Context

No response

vijayank88 commented 1 year ago

@vishallparmar25 How much RAM your system has?

vijayank88 commented 1 year ago

Attach reproducible test case by running make detail_route_issue and upload here.

vishallparmar25 commented 1 year ago

@vishallparmar25 How much RAM your system has?

16 GB RAM

vijayank88 commented 1 year ago

So it is RAM issue.

Command terminated by signal 9
Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.

Once it is reached max RAM (15.5GB) it is got killed. Increase swap memory and try again

vishallparmar25 commented 1 year ago

Attach reproducible test case by running make detail_route_issue and upload here.

It's not possible to share entire tar file here it's large in size. Any specific file you are looking from that tar ?

vishallparmar25 commented 1 year ago

So it is RAM issue.

Command terminated by signal 9
Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.

Once it is reached max RAM (15.5GB) it is got killed. Increase swap memory and try again

Can you please share steps to increase swap memory, or any link. How much it requires additional memory ?

vijayank88 commented 1 year ago

check this link based on OS: https://www.makeuseof.com/swap-files-in-linux/ https://www.tomshardware.com/news/how-to-manage-virtual-memory-pagefile-windows-10,36929.html

rovinski commented 1 year ago

@vishallparmar25 You should try reducing the number of threads before increasing your swap size. The memory used is proportional to the number of threads in DRT.

You can do so by running make NUM_CORES=x where x is a positive integer.

vijayank88 commented 1 year ago

@vishallparmar25 Are you able to complete the routing?

vishallparmar25 commented 1 year ago

@vishallparmar25 Are you able to complete the routing?

No, it's giving same issue, even after increasing 4 Gb of swap memory.

vijayank88 commented 1 year ago

Can you attach following files?

5_1_grt.odb
4_cts.sdc

I will update you how much RAM its required to complete routing.

vishallparmar25 commented 1 year ago

@vishallparmar25 Are you able to complete the routing?

Fired one run with 32Gb swap memory + 16 Gb of RAM. Once it will completed i will update you.

rovinski commented 1 year ago

@vishallparmar25 did you try running with fewer threads yet? Increasing swap is a much worse and much slower solution.

vishallparmar25 commented 1 year ago

@vishallparmar25 did you try running with fewer threads yet? Increasing swap is a much worse and much slower solution. Yes, In that case it also failing.

vijayank88 commented 1 year ago

@vishallparmar25 are you working on confidential design? Still not shared any test case, hence it is difficult to provide the solution.

vishallparmar25 commented 1 year ago

@vishallparmar25 are you working on confidential design? Still not shared any test case, hence it is difficult to provide the solution.

Yes, i am working on confidential design, that's why.

By the way similar kind of issues i am facing in few more design. only similarity is their cell count are bit higher like 1M -3M cell count. Does there any upper limit for instance count or cell count for OpenROAD-flow-script ?

vijayank88 commented 1 year ago

No limitation for instance count. Try with better hardware or try with cloud resources for short-term to complete routing.

vishallparmar25 commented 1 year ago

No limitation for instance count. Try with better hardware or try with cloud resources for short-term to complete routing. Thanks for let me know about no upper limit for cell count or instance count. I am running design with cloud resources only. Below are my cloud configurations.

image

image

vijayank88 commented 1 year ago

For 3M instance count, the above hardware is not enough as per my knowledge. One hint you can check during detailed routing how much memory it is taking for routing, if it is getting signal 9 or child killed you have to increase your RAM. Detailed route log format shown below.

[INFO DRT-0267] cpu time = 00:07:41, elapsed time = 00:00:28, memory = 1459.91 (MB), peak = 2071.53 (MB)

So check Peak Memory for your design routing and when it fails and what its peak memory.

rovinski commented 1 year ago

Yes, 16 GB is no where near enough for a 1-3M inst design. I would estimate somewhere between 48-96 GB minimum.

vishallparmar25 commented 1 year ago

Yes, 16 GB is no where near enough for a 1-3M inst design. I would estimate somewhere between 48-96 GB minimum.

Can you give me minimum system requirement (Number of Processor, RAM, ROM, etc.) for 3M instance count design.

vijayank88 commented 1 year ago

Currently our CI runs with 32 core/128GB RAM. Maybe try with same configuration.

rovinski commented 1 year ago

Currently our CI runs with 32 core/128GB RAM. Maybe try with same configuration.

However, I also don't think that we run with any 3M instance designs in the CI.

Can you give me minimum system requirement (Number of Processor, RAM, ROM, etc.) for 3M instance count design.

I can't give your requirements, but I can give estimates. It depends on your specific design. We have loose estimates for scaling, which I have given based on a 1M instance design. A 3M instance design would probably be closer to 144-256 GB of memory.

It is usually not recommended to run designs that large as a single block. You should look into partitioning your design into smaller, more manageable block sizes.