Open lakshmikanthgummadi opened 11 months ago
vijayank88
commented
11 months ago @lakshmikanthgummadi
Run make cts_issue for your design and attach a test case to debug the same.
Maybe 2M instances, the given RAM is not sufficient. It should terminated the run, don't understand why it is still running.
Just run and check htop what are the jobs running still.
lakshmikanthgummadi
commented
11 months ago killed the run manually !
vvbandeira
commented
11 months ago @lakshmikanthgummadi
We have anecdotal information that a design with 1.7M instances takes over 30h to run on a 32 CPU 128GB RAM machine.
Looks like you have a 4 CPU, and 16GB RAM machine, which might not be sufficient to run such a large design. You have another 32 GB of swap, but if the design uses swap, the run time can be exponentially worse because this will use HDD/SSD as RAM, which is very slow compared to native RAM.
From your last print of htop, was openroad still running? There's no openroad info on the print, and the load average points to no process running for the last past 15 min. If this was before you killed the process, this is useful; otherwise, not so much.
maliberty
commented
11 months ago Is it possible to provide a test case?
lakshmikanthgummadi
commented
11 months ago Again fired the run yesterday its been more then 19 hours , and still it is running on cts stage , sharing the log screen shot and htop
vijayank88
commented
11 months ago @lakshmikanthgummadi Run
make cts_issuefor your design and attach a test case to debug the same. Maybe 2M instances, the given RAM is not sufficient. It should terminated the run, don't understand why it is still running. Just run and checkhtopwhat are the jobs running still.
@lakshmikanthgummadi Please understand few things it is mandatory to get support.
By sharing log/screenshot won't resolve your issue.
Run make cts_issue and attach test case generated in flow directory to get solution.
rovinski
commented
11 months ago @lakshmikanthgummadi a 2M instance design will not finish on a machine with 16GB RAM. Full stop. You need on the order of 8 times more RAM. Swap will not help you. Your only other option is to partition the design down to much much less than 2M instances per block.
lakshmikanthgummadi
commented
11 months ago @lakshmikanthgummadi Run
make cts_issuefor your design and attach a test case to debug the same. Maybe 2M instances, the given RAM is not sufficient. It should terminated the run, don't understand why it is still running. Just run and checkhtopwhat are the jobs running still.@lakshmikanthgummadi Please understand few things it is mandatory to get support. By sharing log/screenshot won't resolve your issue. Run
make cts_issueand attach test case generated inflowdirectory to get solution.
tar file size was 1113 mb, not allowed to upload here , how to share the file ??
vijayank88
commented
11 months ago Upload it to google drive and share access link
lakshmikanthgummadi
commented
11 months ago Upload it to google drive and share access link
https://drive.google.com/file/d/1OfY6tSnXlllg7-TJEaHfYeYHqPxLMhhn/view?usp=sharing
vijayank88
commented
11 months ago Upload it to google drive and share access link
https://drive.google.com/file/d/1OfY6tSnXlllg7-TJEaHfYeYHqPxLMhhn/view?usp=sharing
provide public access to file location
lakshmikanthgummadi
commented
11 months ago Upload it to google drive and share access link
https://drive.google.com/file/d/1OfY6tSnXlllg7-TJEaHfYeYHqPxLMhhn/view?usp=sharing
provide public access to file location
given !!
lakshmikanthgummadi
commented
11 months ago Any Update @ @vijayank88 @rovinski @
vishallparmar25
commented
11 months ago I am also facing same issue, any update ???
vijayank88
commented
11 months ago @lakshmikanthgummadi
Can you share your RTL, config.mk, constraint.sdc?
@vishallparmar25 Open a new issue by following issue template and attach a test case.
maliberty
commented
11 months ago Does this make sense:
[WARNING RSZ-0069] skipping net u_core/u_icache/data_addr_r[0] with 103883 pins.Is that single bit really a global signal?
maliberty
commented
11 months ago Something is quite wrong in 3_4_resizer.log
==========================================================================
resizer report_wns
--------------------------------------------------------------------------
wns -164492.39
...
resizer report_check_types -max_slew -max_cap -max_fanout -violators
--------------------------------------------------------------------------
max slew
Pin Limit Slew Slack
------------------------------------------------------------
u_core/u_icache/_0848_/Y 320.00 252488.52 -252168.52 (VIOLATED)
...
max capacitance
Pin Limit Cap Slack
------------------------------------------------------------
u_core/u_icache/_0848_/Y 92.16 84234.69 -84142.52 (VIOLATED)There are huge violations. In CTS the timing optimization is starting from a horrible place. I think we need to understand what is happening in 3_4 first.
maliberty
commented
11 months ago The 3_4 issue comes back to the warning above. Such a large net produces terrible runtimes in building a Steiner tree and so we skip it. Please provide more detail on what the intention is.
maliberty
commented
11 months ago It looks like you have rams that you are synthesizing at gate level (eg icache_data_ram). You should use a RAM macro for those.
vijayank88
commented
11 months ago @lakshmikanthgummadi Can you update status on this?
lakshmikanthgummadi
commented
11 months ago @lakshmikanthgummadi Can you update status on this?
vijayank88
commented
11 months ago @lakshmikanthgummadi Have you replaced with RAM or how you resolved the issue?
lakshmikanthgummadi
commented
11 months ago @lakshmikanthgummadi Have you replaced with RAM or how you resolved the issue?
we are migrating to new cloud server once it is done i will update , It may take one week of time ! Thanks
Subject
Describe the bug
ORFS CTS taking around 6 days and still it's running, working on 2M instance count design with cloud resource, cloud configs are below
below are RAM Config
Is there any possibility that we can shorten runtime??
Expected Behavior
CTS has to finish within four hours !
Environment
To Reproduce
set env(RCX_RULES) "./platforms/asap7/rcx_patterns.rules" set env(DESIGN_NAME) "riscv_soc" set env(SDC_FILE) "./designs/asap7/riscv_soc/constraint.sdc" set env(IO_PLACER_H) "M2 M4 M6 M8" set env(GDS_FILES) "./platforms/asap7/gds/asap7sc7p5t_28_R_220121a.gds" set env(MAKE_TRACKS) "./platforms/asap7/openRoad/make_tracks.tcl" set env(NPROC) "4" set env(TAP_CELL_NAME) "TAPCELL_ASAP7_75t_R" set env(HIER_REPORT_SCRIPT) "./scripts/synth_hier_report.tcl" set env(RTLMP_FLOW) "True" set env(CTS_BUF_CELL) "BUFx4_ASAP7_75t_R" set env(STREAM_SYSTEM) "GDS" set env(SYNTH_HIERARCHICAL) "1" set env(LATCH_MAP_FILE) "./platforms/asap7/yoSys/cells_latch_R.v" set env(TC_VOLTAGE) "0.70" set env(WC_DFF_LIB_FILE) "./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_SS_nldm_220123.lib" set env(TAPCELL_TCL) "./platforms/asap7/openRoad/tapcell.tcl" set env(BC_VOLTAGE) "0.77" set env(VOLTAGE) "0.77" set env(TCLLIBPATH) "util/cell-veneer" set env(GPL_ROUTABILITY_DRIVEN) "1" set env(WC_TEMPERATURE) "100C" set env(WC_LIB_FILES) "./platforms/asap7/lib/asap7sc7p5t_AO_RVT_SS_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_SS_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_SS_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_SS_nldm_220123.lib ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_SS_nldm_211120.lib.gz" set env(OBJECTS_DIR) "./objects/asap7/riscv_soc/base" set env(DESIGN_NICKNAME) "riscv_soc" set env(ENABLE_DPO) "1" set env(BC_TEMPERATURE) "25C" set env(HAS_IO_CONSTRAINTS) "1" set env(YOSYS_CMD) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/yosys/bin/yosys" set env(RTLMP_RPT_DIR) "./objects/asap7/riscv_soc/base/rtlmp" set env(SYNTH_SCRIPT) "./scripts/synth.tcl" set env(ABC_DRIVER_CELL) "BUFx2_ASAP7_75t_R" set env(PWR_NETS_VOLTAGES) "VDD 0.77" set env(BC_LIB_FILES) "./platforms/asap7/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib" set env(CURDIR) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/flow" set env(TIELO_CELL_AND_PORT) "TIELOx1_ASAP7_75t_R L" set env(DESIGN_HOME) "./designs" set env(SCRIPTS_DIR) "./scripts" set env(IO_PLACER_V) "M3 M5 M7 M9" set env(MAX_UNGROUP_SIZE) "0" set env(RESYNTH_TIMING_RECOVER) "0" set env(SYNTH_ARGS) "-flatten" set env(ABC_AREA) "1" set env(CTS_BUF_DISTANCE) "60" set env(LSORACLE_PLUGIN) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/yosys/share/yosys/plugin/oracle.so" set env(LOG_DIR) "./logs/asap7/riscv_soc/base" set env(CELL_PAD_IN_SITES_DETAIL_PLACEMENT) "2" set env(TECH_LEF) "./platforms/asap7/lef/asap7_tech_1x_201209.lef" set env(GND_NETS_VOLTAGES) "VSS 0.0" set env(DIE_AREA) "0 0 1600 1700" set env(RESYNTH_AREA_RECOVER) "0" set env(ISSUE_SCRIPTS) "add_routing_blk cdl cts deleteNonClkNets deletePowerNets deleteRoutingObstructions density_fill detail_place detail_route fillcell final_report floorplan generate_abstract global_place global_place_skip_io global_route gui io_placement io_placement_random klayout load macro_place pdn placement_blockages read_liberty read_macro_placement report_metrics resize run_all save_images synth synth_hier_report synth_preamble tapcell tdms_place view_cells write_def write_ref_sdc write_verilog yosys" set env(PLATFORM) "asap7" set env(LIB_FILES) "./platforms/asap7/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib" set env(SET_RC_TCL) "./platforms/asap7/setRC.tcl" set env(GALLERY_REPORT) "0" set env(SHELL) "/usr/bin/env bash" set env(FILL_CELLS) "FILLERxp5_ASAP7_75t_R FILLER_ASAP7_75t_R DECAPx1_ASAP7_75t_R DECAPx2_ASAP7_75t_R DECAPx4_ASAP7_75t_R DECAPx6_ASAP7_75t_R DECAPx10_ASAP7_75t_R" set env(OPENROAD_NO_EXIT_CMD) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -no_init" set env(OPENROAD_EXE) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad" set env(TNS_END_PERCENT) "5" set env(GPL_TIMING_DRIVEN) "1" set env(CELL_PAD_IN_SITES_GLOBAL_PLACEMENT) "4" set env(MIN_BUF_CELL_AND_PORTS) "BUFx2_ASAP7_75t_R A Y" set env(PLACE_DENSITY) "0.1" set env(RESULTS_DIR) "./results/asap7/riscv_soc/base" set env(STDBUF_CMD) "stdbuf -o L" set env(CLKGATE_MAP_FILE) "./platforms/asap7/yoSys/cells_clkgate_R.v" set env(REPORTS_DIR) "./reports/asap7/riscv_soc/base" set env(PLATFORM_DIR) "./platforms/asap7" set env(PLATFORM_HOME) "./platforms" set env(GDSOAS_FILES) "./platforms/asap7/gds/asap7sc7p5t_28_R_220121a.gds" set env(LSORACLE_KAHYPAR_CONFIG) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/LSOracle/share/lsoracle/test.ini" set env(WORK_HOME) "." set env(TIEHI_CELL_AND_PORT) "TIEHIx1_ASAP7_75t_R H" set env(DESIGN_CONFIG) "./designs/asap7/riscv_soc/config.mk" set env(DONT_USE_SC_LIB) "./objects/asap7/riscv_soc/base/lib/merged.lib" set env(BC_DFF_LIB_FILE) "./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib" set env(CORE_AREA) "0.05 0.05 1599.95 1699.95" set env(STREAM_SYSTEM_EXT) "gds" set env(TC_TEMPERATURE) "0C" set env(ADDER_MAP_FILE) "./platforms/asap7/yoSys/cells_adders_R.v" set env(LSORACLE_CMD) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/LSOracle/bin/lsoracle" set env(PLACE_SITE) "asap7sc7p5t" set env(YOSYS_FLAGS) "-v 3" set env(SYNTH_STOP_MODULE_SCRIPT) "./objects/asap7/riscv_soc/base/mark_hier_stop_modules.tcl" set env(DESIGN_DIR) "./designs/asap7/riscv_soc/" set env(TEMPERATURE) "25C" set env(TC_LIB_FILES) "./platforms/asap7/lib/asap7sc7p5t_AO_RVT_TT_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_INVBUF_RVT_TT_nldm_220122.lib.gz ./platforms/asap7/lib/asap7sc7p5t_OA_RVT_TT_nldm_211120.lib.gz ./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_TT_nldm_220123.lib ./platforms/asap7/lib/asap7sc7p5t_SIMPLE_RVT_TT_nldm_211120.lib.gz" set env(FLOW_HOME) "." set env(MACRO_PLACE_CHANNEL) "12 12" set env(DONT_USE_CELLS) "x1p_ASAP7 xp_ASAP7 SDF ICG DFFH" set env(SC_LEF) "./platforms/asap7/lef/asap7sc7p5t_28_R_1x_220121a.lef" set env(RESULTS_ODB) "2_1_floorplan.odb 2_2_floorplan_io.odb 2_3_floorplan_tdms.odb 2_4_floorplan_macro.odb 2_5_floorplan_tapcell.odb 2_6_floorplan_pdn.odb 2_floorplan.odb 3_1_place_gp_skip_io.odb 3_2_place_iop.odb 3_3_place_gp.odb 3_4_place_resized.odb 3_5_place_dp.odb 3_place.odb" set env(TEST_DIR) "./test" set env(MACRO_PLACE_HALO) "10 10" set env(DONT_USE_LIBS) " ./objects/asap7/riscv_soc/base/lib/asap7sc7p5t_AO_RVT_FF_nldm_211120.lib ./objects/asap7/riscv_soc/base/lib/asap7sc7p5t_INVBUF_RVT_FF_nldm_220122.lib ./objects/asap7/riscv_soc/base/lib/asap7sc7p5t_OA_RVT_FF_nldm_211120.lib ./objects/asap7/riscv_soc/base/lib/asap7sc7p5t_SIMPLE_RVT_FF_nldm_211120.lib ./objects/asap7/riscv_soc/base/lib/asap7sc7p5t_SEQ_RVT_FF_nldm_220123.lib" set env(PUBLIC) "nangate45 sky130hd sky130hs asap7" set env(WRAP_CFG) "./platforms/asap7/wrapper.cfg" set env(DPO_MAX_DISPLACEMENT) "5 1" set env(GDS_FINAL_FILE) "./results/asap7/riscv_soc/base/6_final.gds" set env(VERILOG_FILES) "./designs/src/riscv_soc/axi4_arb.v ./designs/src/riscv_soc/axi4_lite_tap.v ./designs/src/riscv_soc/axi4_retime.v ./designs/src/riscv_soc/axi4lite_axi4_conv.v ./designs/src/riscv_soc/dport_bridge.v ./designs/src/riscv_soc/gpio.v ./designs/src/riscv_soc/gpio_defs.v ./designs/src/riscv_soc/icache.v ./designs/src/riscv_soc/icache_data_ram.v ./designs/src/riscv_soc/icache_tag_ram.v ./designs/src/riscv_soc/irq_ctrl.v ./designs/src/riscv_soc/irq_ctrl_defs.v ./designs/src/riscv_soc/riscv_alu.v ./designs/src/riscv_soc/riscv_core.v ./designs/src/riscv_soc/riscv_csr.v ./designs/src/riscv_soc/riscv_decode.v ./designs/src/riscv_soc/riscv_defs.v ./designs/src/riscv_soc/riscv_exec.v ./designs/src/riscv_soc/riscv_fetch.v ./designs/src/riscv_soc/riscv_lsu.v ./designs/src/riscv_soc/riscv_muldiv.v ./designs/src/riscv_soc/riscv_regfile.v ./designs/src/riscv_soc/riscv_soc.v ./designs/src/riscv_soc/riscv_top.v ./designs/src/riscv_soc/soc.v ./designs/src/riscv_soc/spi_lite.v ./designs/src/riscv_soc/spi_lite_defs.v ./designs/src/riscv_soc/timer.v ./designs/src/riscv_soc/timer_defs.v ./designs/src/riscv_soc/uart_lite.v ./designs/src/riscv_soc/uart_lite_defs.v" set env(PDN_TCL) "./platforms/asap7/openRoad/pdn/grid_strategy-M1-M2-M5-M6.tcl" set env(TIME_TEST) "foo" set env(UTILS_DIR) "./util" set env(ABC_CLOCK_PERIOD_IN_PS) "2000" set env(MIN_ROUTING_LAYER) "M2" set env(PROCESS) "7" set env(GDS_MERGED_FILE) "./results/asap7/riscv_soc/base/6_1_merged.gds" set env(TEST_SCRIPT) "./test/core_tests.sh" set env(NUM_CORES) "4" set env(OPENROAD_CMD) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -exit -no_init" set env(ABC_LOAD_IN_FF) "3.898" set env(MAX_ROUTING_LAYER) "M9" set env(CORNER) "BC" set env(OPENROAD_ARGS) "-no_init" set env(OPENROAD_GUI_CMD) "/home/ubuntu/Opensource/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad -gui" set env(WC_VOLTAGE) "0.63" set env(PLACE_PINS_ARGS) "-min_distance 20 -min_distance_in_tracks" set env(TC_DFF_LIB_FILE) "./platforms/asap7/lib/asap7sc7p5t_SEQ_RVT_TT_nldm_220123.lib" set env(FLOW_VARIANT) "base" set env(RULES_DESIGN) "./designs/asap7/riscv_soc/rules-base.json" set env(RTLMP_BLOCKAGE_FILE) "./objects/asap7/riscv_soc/base/rtlmp/partition.txt.blockage" set env(RTLMP_RPT_FILE) "partition.txt" set env(IR_DROP_LAYER) "M1" set env(GDS_ALLOW_EMPTY) "fakeram."
Relevant log output
No response
Screenshots
Additional Context
No response