Open vishallparmar25 opened 1 year ago
@vishallparmar25 How much RAM your system has?
Attach reproducible test case by running make detail_route_issue
and upload here.
@vishallparmar25 How much RAM your system has?
16 GB RAM
So it is RAM issue.
Command terminated by signal 9
Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.
Once it is reached max RAM (15.5GB) it is got killed. Increase swap memory and try again
Attach reproducible test case by running
make detail_route_issue
and upload here.
It's not possible to share entire tar file here it's large in size. Any specific file you are looking from that tar ?
So it is RAM issue.
Command terminated by signal 9 Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.
Once it is reached max RAM (15.5GB) it is got killed. Increase swap memory and try again
Can you please share steps to increase swap memory, or any link. How much it requires additional memory ?
@vishallparmar25 You should try reducing the number of threads before increasing your swap size. The memory used is proportional to the number of threads in DRT.
You can do so by running make NUM_CORES=x
where x
is a positive integer.
@vishallparmar25 Are you able to complete the routing?
@vishallparmar25 Are you able to complete the routing?
No, it's giving same issue, even after increasing 4 Gb of swap memory.
Can you attach following files?
5_1_grt.odb
4_cts.sdc
I will update you how much RAM its required to complete routing.
@vishallparmar25 Are you able to complete the routing?
Fired one run with 32Gb swap memory + 16 Gb of RAM. Once it will completed i will update you.
@vishallparmar25 did you try running with fewer threads yet? Increasing swap is a much worse and much slower solution.
@vishallparmar25 did you try running with fewer threads yet? Increasing swap is a much worse and much slower solution. Yes, In that case it also failing.
@vishallparmar25 are you working on confidential design? Still not shared any test case, hence it is difficult to provide the solution.
@vishallparmar25 are you working on confidential design? Still not shared any test case, hence it is difficult to provide the solution.
Yes, i am working on confidential design, that's why.
By the way similar kind of issues i am facing in few more design. only similarity is their cell count are bit higher like 1M -3M cell count. Does there any upper limit for instance count or cell count for OpenROAD-flow-script ?
No limitation for instance count. Try with better hardware or try with cloud resources for short-term to complete routing.
No limitation for instance count. Try with better hardware or try with cloud resources for short-term to complete routing. Thanks for let me know about no upper limit for cell count or instance count. I am running design with cloud resources only. Below are my cloud configurations.
For 3M instance count, the above hardware is not enough as per my knowledge.
One hint you can check during detailed routing how much memory it is taking for routing, if it is getting signal 9
or child killed
you have to increase your RAM. Detailed route log format shown below.
[INFO DRT-0267] cpu time = 00:07:41, elapsed time = 00:00:28, memory = 1459.91 (MB), peak = 2071.53 (MB)
So check Peak Memory
for your design routing and when it fails and what its peak memory.
Yes, 16 GB is no where near enough for a 1-3M inst design. I would estimate somewhere between 48-96 GB minimum.
Yes, 16 GB is no where near enough for a 1-3M inst design. I would estimate somewhere between 48-96 GB minimum.
Can you give me minimum system requirement (Number of Processor, RAM, ROM, etc.) for 3M instance count design.
Currently our CI runs with 32 core/128GB RAM. Maybe try with same configuration.
Currently our CI runs with 32 core/128GB RAM. Maybe try with same configuration.
However, I also don't think that we run with any 3M instance designs in the CI.
Can you give me minimum system requirement (Number of Processor, RAM, ROM, etc.) for 3M instance count design.
I can't give your requirements, but I can give estimates. It depends on your specific design. We have loose estimates for scaling, which I have given based on a 1M instance design. A 3M instance design would probably be closer to 144-256 GB of memory.
It is usually not recommended to run designs that large as a single block. You should look into partitioning your design into smaller, more manageable block sizes.
Subject
[Stage]: Detail Router.
Describe the bug
ORFS flow failed at detail route stage
[INFO DRT-0036] Pad guide region query size = 0. [INFO DRT-0179] Init gr pin query. [INFO DRT-0185] Post process initialize RPin region query. [INFO DRT-0181] Start track assignment. [INFO DRT-0184] Done with 726268 vertical wires in 15 frboxes and 499275 horizontal wires in 15 frboxes. [INFO DRT-0186] Done with 55361 vertical wires in 15 frboxes and 81996 horizontal wires in 15 frboxes. [INFO DRT-0182] Complete track assignment. [INFO DRT-0267] cpu time = 00:03:26, elapsed time = 00:01:00, memory = 5321.84 (MB), peak = 6472.33 (MB) [INFO DRT-0187] Start routing data preparation. [INFO DRT-0267] cpu time = 00:00:01, elapsed time = 00:00:01, memory = 5321.84 (MB), peak = 6472.33 (MB) [INFO DRT-0194] Start detail routing. [INFO DRT-0195] Start 0th optimization iteration. Command terminated by signal 9 Elapsed time: 3:26.31[h:]min:sec. CPU time: user 681.69 sys 21.72 (340%). Peak memory: 15534688KB.
Expected Behavior
[INFO DRT-0198] Complete detail routing. Total wire length = 127688 um. Total wire length on LAYER M1 = 0 um.
Environment
To Reproduce
export RCX_RULES='./platforms/asap7/rcx_patterns.rules' export DESIGN_NAME='tpu_top' export SDC_FILE='./designs/asap7/TPU_TOP/constraint.sdc' export IO_PLACER_H='M2 M4 M6 M8' export GDS_FILES='./platforms/asap7/gds/asap7sc7p5t_28_R_220121a.gds' export MAKE_TRACKS='./platforms/asap7/openRoad/make_tracks.tcl' export NPROC='4' export TAP_CELL_NAME='TAPCELL_ASAP7_75t_R' export HIER_REPORT_SCRIPT='./scripts/synth_hier_report.tcl' export RTLMP_FLOW='True' export CTS_BUF_CELL='BUFx4_ASAP7_75t_R' export STREAM_SYSTEM='GDS' export SYNTH_HIERARCHICAL='1' export LATCH_MAP_FILE='./platforms/asap7/yoSys/cells_latch_R.v' export TC_VOLTAGE='0.70' export WC_DFF_LIB_FILE='./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_SS_nldm_220123.lib' export TAPCELL_TCL='./platforms/asap7/openRoad/tapcell.tcl' export BC_VOLTAGE='0.77' export VOLTAGE='0.77' export TCLLIBPATH='util/cell-veneer' export GPL_ROUTABILITY_DRIVEN='1' export WC_TEMPERATURE='100C' export WC_LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_SS_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_SS_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_SS_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_SS_nldm_220123.lib ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_SS_nldm_211120.lib.gz' export OBJECTS_DIR='./objects/asap7/TPU_TOP/base' export DESIGN_NICKNAME='TPU_TOP' export ENABLE_DPO='1' export BC_TEMPERATURE='25C' export HAS_IO_CONSTRAINTS='1' export YOSYS_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys' export RTLMP_RPT_DIR='./objects/asap7/TPU_TOP/base/rtlmp' export SYNTH_SCRIPT='./scripts/synth.tcl' export ABC_DRIVER_CELL='BUFx2_ASAP7_75t_R' export PWR_NETS_VOLTAGES='"VDD 0.77"' export BC_LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export CURDIR='/home/ubuntu/Opensource/OpenROAD-flow-scripts/flow' export TIELO_CELL_AND_PORT='TIELOx1_ASAP7_75t_R L' export DESIGN_HOME='./designs' export SCRIPTS_DIR='./scripts' export IO_PLACER_V='M3 M5 M7 M9' export MAX_UNGROUP_SIZE='0' export RESYNTH_TIMING_RECOVER='0' export SYNTH_ARGS='-flatten' export ABC_AREA='1' export CTS_BUF_DISTANCE='60' export LSORACLE_PLUGIN='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/yosys/share/yosys/plugin/oracle.so' export LOG_DIR='./logs/asap7/TPU_TOP/base' export CELL_PAD_IN_SITES_DETAIL_PLACEMENT='2' export TECH_LEF='./platforms/asap7/lef/asap7_tech_1x_201209.lef' export GND_NETS_VOLTAGES='"VSS 0.0"' export DIE_AREA='0 0 400 400' export RESYNTH_AREA_RECOVER='0' export ISSUE_SCRIPTS='add_routing_blk cdl cts deleteNonClkNets deletePowerNets deleteRoutingObstructions density_fill detail_place detail_route fillcell final_report floorplan generate_abstract global_place global_place_skip_io global_route gui io_placement io_placement_random klayout load macro_place pdn placement_blockages read_liberty read_macro_placement report_metrics resize run_all save_images synth synth_hier_report synth_preamble tapcell tdms_place view_cells write_def write_ref_sdc write_verilog yosys' export PLATFORM='asap7' export LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export SET_RC_TCL='./platforms/asap7/setRC.tcl' export GALLERY_REPORT='0' export SHELL='/usr/bin/env bash' export FILL_CELLS='FILLERxp5_ASAP7_75t_R FILLER_ASAP7_75t_R DECAPx1_ASAP7_75t_R DECAPx2_ASAP7_75t_R DECAPx4_ASAP7_75t_R DECAPx6_ASAP7_75t_R DECAPx10_ASAP7_75t_R' export OPENROAD_NO_EXIT_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -no_init' export OPENROAD_EXE='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad' export TNS_END_PERCENT='5' export GPL_TIMING_DRIVEN='1' export CELL_PAD_IN_SITES_GLOBAL_PLACEMENT='4' export MIN_BUF_CELL_AND_PORTS='BUFx2_ASAP7_75t_R A Y' export PLACE_DENSITY='0.25' export RESULTS_DIR='./results/asap7/TPU_TOP/base' export STDBUF_CMD='stdbuf -o L' export CLKGATE_MAP_FILE='./platforms/asap7/yoSys/cells_clkgate_R.v' export REPORTS_DIR='./reports/asap7/TPU_TOP/base' export PLATFORM_DIR='./platforms/asap7' export PLATFORM_HOME='./platforms' export GDSOAS_FILES='./platforms/asap7/gds/asap7sc7p5t_28_R_220121a.gds' export LSORACLE_KAHYPAR_CONFIG='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/LSOracle/share/lsoracle/test.ini' export WORK_HOME='.' export TIEHI_CELL_AND_PORT='TIEHIx1_ASAP7_75t_R H' export DESIGN_CONFIG='./designs/asap7/TPU_TOP/config.mk' export DONT_USE_SC_LIB='./objects/asap7/TPU_TOP/base/lib/merged.lib' export BC_DFF_LIB_FILE='./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export CORE_AREA='0.05 0.05 399.95 399.95' export STREAM_SYSTEM_EXT='gds' export TC_TEMPERATURE='0C' export ADDER_MAP_FILE='./platforms/asap7/yoSys/cells_adders_R.v' export LSORACLE_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/LSOracle/bin/lsoracle' export PLACE_SITE='asap7sc7p5t' export YOSYS_FLAGS='-v 3' export SYNTH_STOP_MODULE_SCRIPT='./objects/asap7/TPU_TOP/base/mark_hier_stop_modules.tcl' export DESIGN_DIR='./designs/asap7/TPU_TOP/' export TEMPERATURE='25C' export TC_LIB_FILES='./platforms/asap7/lib/asap7sc7p5t_AO_RVT_TT_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_TT_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_TT_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_TT_nldm_220123.lib ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_TT_nldm_211120.lib.gz' export FLOW_HOME='.' export MACRO_PLACE_CHANNEL='12 12' export DONT_USE_CELLS='x1p_ASAP7 xp_ASAP7 SDF ICG DFFH' export SC_LEF='./platforms/asap7/lef/asap7sc7p5t_28_R_1x_220121a.lef' export RESULTS_ODB='2_1_floorplan.odb 2_2_floorplan_io.odb 2_3_floorplan_tdms.odb 2_4_floorplan_macro.odb 2_5_floorplan_tapcell.odb 2_6_floorplan_pdn.odb 2_floorplan.odb 3_1_place_gp_skip_io.odb 3_2_place_iop.odb 3_3_place_gp.odb 3_4_place_resized.odb 3_5_place_dp.odb 3_place.odb 4_1_cts.odb 4_2_cts_fillcell.odb 4_cts.odb 5_1_grt.odb' export TEST_DIR='./test' export MACRO_PLACE_HALO='10 10' export DONT_USE_LIBS=' ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib ./objects/asap7/TPU_TOP/base/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib' export PUBLIC='nangate45 sky130hd sky130hs asap7' export WRAP_CFG='./platforms/asap7/wrapper.cfg' export DPO_MAX_DISPLACEMENT='5 1' export GDS_FINAL_FILE='./results/asap7/TPU_TOP/base/6_final.gds' export VERILOG_FILES='./designs/src/TPU_TOP/addr_sel.v ./designs/src/TPU_TOP/quantize.v ./designs/src/TPU_TOP/sram_256x32b.v ./designs/src/TPU_TOP/sram_64x512b.v ./designs/src/TPU_TOP/systolic.v ./designs/src/TPU_TOP/systolic_control.v ./designs/src/TPU_TOP/tpu_top.v ./designs/src/TPU_TOP/write_out.v' export PDN_TCL='./platforms/asap7/openRoad/pdn/grid_strategy-M1-M2-M5-M6.tcl' export TIME_TEST='foo' export UTILS_DIR='./util' export ABC_CLOCK_PERIOD_IN_PS='1000' export MIN_ROUTING_LAYER='M2' export PROCESS='7' export GDS_MERGED_FILE='./results/asap7/TPU_TOP/base/6_1_merged.gds' export TEST_SCRIPT='./test/core_tests.sh' export NUM_CORES='4' export OPENROAD_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -exit -no_init' export ABC_LOAD_IN_FF='3.898' export MAX_ROUTING_LAYER='M9' export CORNER='BC' export OPENROAD_ARGS='-no_init' export OPENROAD_GUI_CMD='/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -gui' export WC_VOLTAGE='0.63' export PLACE_PINS_ARGS='-min_distance 20 -min_distance_in_tracks' export TC_DFF_LIB_FILE='./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_TT_nldm_220123.lib' export FLOW_VARIANT='base' export RULES_DESIGN='./designs/asap7/TPU_TOP/rules-base.json' export RTLMP_BLOCKAGE_FILE='./objects/asap7/TPU_TOP/base/rtlmp/partition.txt.blockage' export RTLMP_RPT_FILE='partition.txt' export IR_DROP_LAYER='M1' export GDS_ALLOW_EMPTY='fakeram.'
Relevant log output
Screenshots
config.mk
Instance Count
Placement Density heat map
Routing congestion heat map
Additional Context
No response