search

The-OpenROAD-Project / OpenROAD

OpenROAD's unified application implementing an RTL-to-GDS Flow. Documentation at https://openroad.readthedocs.io/en/latest/
https://theopenroadproject.org/
BSD 3-Clause "New" or "Revised" License
1.6k stars 556 forks source link

Post-GRT, resizer fails with cryptic error #5590

Closed donn closed 2 months ago

donn commented 3 months ago

Describe the bug

Running post-GRT timing optimizations fails with a cryptic error and later a stack dump (see log), but only in the event that the database has been written after global routing, OpenROAD exited out of, then the database is re-read.

If GRT and resizing happens in the same script, this works fine. However, this may be a waste of compute for some designs where GRT takes a non-trivial amount of time.

Expected Behavior

Whatever data global routing saves should be in Odb/more actionable error

Environment

Git commit: 14b1cbdf0c3160dd34ff79cf413d1f66f9ec4eac
kernel: Darwin 23.5.0
os: macOS 14.5
cmake version 3.29.2
CMake Warning (dev) at src/sta/CMakeLists.txt:32 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'USE_TCL_READLINE'.
This warning is for project developers.  Use -Wno-dev to suppress it.

Number of processor cores: 24
CMake Warning:
  Manually-specified variables were not used by the project:

    VERBOSE

-- The CXX compiler identification is Clang 16.0.6
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/c1ib07kj7baid3s3wf5m98hcpbvgs8mk-clang-wrapper-16.0.6/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-15221-g14b1cbdf0
-- System name: Darwin
-- Compiler: Clang 16.0.6
-- Build type: RELEASE
-- Install prefix: /var/empty/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- The C compiler identification is Clang 16.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/c1ib07kj7baid3s3wf5m98hcpbvgs8mk-clang-wrapper-16.0.6/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Success
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-std=c++17
-- Performing Test C_COMPILER_SUPPORTS__-std=c++17 - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-std=c++17
-- Performing Test CXX_COMPILER_SUPPORTS__-std=c++17 - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/lib/libtcl.dylib
-- TCL header: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/include/tcl.h
-- TCL readline library: /nix/store/0cwgrwkw5kfshdz7bkqjmhm9clas9fqs-tclreadline-2.3.8/lib/libtclreadline.dylib
-- TCL readline header: /nix/store/0cwgrwkw5kfshdz7bkqjmhm9clas9fqs-tclreadline-2.3.8/include
-- Found SWIG: /nix/store/q6jgc1rpam7xqrylwq6gxq2f3arcmhvs-swig-4.0.2/bin/swig (found suitable version "4.0.2", minimum required is "4.0")
-- Found Boost: /nix/store/67hsz87y3shaq2qsql57w5gg8wih8g2m-boost-1.83.0-dev/lib/cmake/Boost-1.83.0/BoostConfig.cmake (found version "1.83.0")
-- boost: 1.83.0
-- Found Python3: /nix/store/0hhzvkw889bsybhqxy12ky4jx6a95p2d-python3-3.11.9/include/python3.11 (found version "3.11.9") found components: Development Development.Module Development.Embed
-- Found ZLIB: /nix/store/5g9k8zyjv029pw87xf95pci414r8pw8j-zlib-1.3.1/lib/libz.dylib (found version "1.3.1")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- spdlog: 1.13.0
-- Found BISON: /nix/store/zz65ns9x0hczhhdq74905dk297z29zvn-bison-3.8.2/bin/bison (found version "3.8.2")
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- STA version: 2.6.0
-- STA git sha: 4fe6f220a7e3fbf18aff5b59aa1c16295f7bb702
-- System name: Darwin
-- Compiler: Clang 16.0.6
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /var/empty/local
-- Found FLEX: /nix/store/n0kkb647sybl2320a1pk85681jbb05p6-flex-2.6.4/bin/flex (found version "2.6.4")
-- TCL library: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/lib/libtcl.dylib
-- TCL header: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/include/tcl.h
-- TCL readline library: /nix/store/0cwgrwkw5kfshdz7bkqjmhm9clas9fqs-tclreadline-2.3.8/lib/libtclreadline.dylib
-- TCL readline header: /nix/store/0cwgrwkw5kfshdz7bkqjmhm9clas9fqs-tclreadline-2.3.8/include/tclreadline.h
-- CUDD library: /nix/store/yklam2c2z7i3msfg106r2v3g46wprjwn-cudd-3.0.0/lib/libcudd.a
-- CUDD header: /nix/store/yklam2c2z7i3msfg106r2v3g46wprjwn-cudd-3.0.0/include/cudd.h
-- SSTA: 0
-- Found SWIG: /nix/store/q6jgc1rpam7xqrylwq6gxq2f3arcmhvs-swig-4.0.2/bin/swig (found suitable version "4.0.2", minimum required is "3.0")
-- STA executable: /Users/donn/efabless/openroad/src/sta/app/sta
-- Found Protobuf: /nix/store/a4n8n8wh80wllzrcgrsi94akgj22417g-protobuf-21.12/lib/libprotobuf.dylib (found version "3.21.12")
-- Found re2: /nix/store/ha95iybvpwxg89q3d7q0453fxccxx6dl-re2-2024-05-01-dev/lib/cmake/re2/re2Config.cmake (found version "11.0.0")
-- Found GLPK: /nix/store/zv2pdp1np8n19qgzk0x10kdp2pwfahja-glpk-5.0/lib/libglpk.dylib
-- Found OpenMP_CXX: -fopenmp=libomp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- TCL library: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/lib/libtcl.dylib
-- TCL header: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/include/tcl.h
-- Found OpenMP_C: -fopenmp=libomp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- GUI is enabled
-- Charts widget is enabled
-- Found Boost: /nix/store/67hsz87y3shaq2qsql57w5gg8wih8g2m-boost-1.83.0-dev/lib/cmake/Boost-1.83.0/BoostConfig.cmake (found version "1.83.0") found components: serialization
-- Could NOT find VTune (missing: VTune_LIBRARIES VTune_INCLUDE_DIRS) 
-- Found Boost: /nix/store/67hsz87y3shaq2qsql57w5gg8wih8g2m-boost-1.83.0-dev/lib/cmake/Boost-1.83.0/BoostConfig.cmake (found suitable version "1.83.0", minimum required is "1.78")
-- TCL library: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/lib/libtcl.dylib
-- TCL header: /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/include/tcl.h
-- Found Boost: /nix/store/67hsz87y3shaq2qsql57w5gg8wih8g2m-boost-1.83.0-dev/lib/cmake/Boost-1.83.0/BoostConfig.cmake (found version "1.83.0") found components: serialization system thread
-- Found Eigen3: /nix/store/2arrr3d4ixycdq5lmpqxydz30pip7h7m-eigen-3.4.0/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0")
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done (27.3s)
-- Generating done (0.8s)
-- Build files have been written to: /tmp/nix-shell.w0nava/tmp.GMzJwUoQd6

To Reproduce

Concatenate the file to extract it.

./run_good.sh is the example with both GRT and RSZ in the same script that works. ./run_bad.sh is the example with GRT and RSZ in separate scripts that fails.

repro.partaa.tgz repro.partab.tgz

Relevant log output

Reading OpenROAD database at './intermediate_bad.odb'…
Reading design constraints file at './files/Users/donn/efabless/caravel_openframe_project/openlane/picosoc/base.sdc'…
[INFO]: Setting output delay to: 22
[INFO]: Setting input delay to: 0
[WARNING STA-0441] set_input_delay relative to a clock defined on the same port/pin not allowed.
[WARNING STA-0441] set_input_delay relative to a clock defined on the same port/pin not allowed.
[WARNING STA-0441] set_input_delay relative to a clock defined on the same port/pin not allowed.
[INFO]: Cap load range: 0.5 : 1.0
[INFO]: Input transition range: 1 : 1.19
[INFO]: Setting derate factor to: 15.0 %
[INFO] Setting RC values…
[INFO RSZ-0094] Found 557 endpoints with setup violations.
[INFO RSZ-0099] Repairing 557 out of 557 (100.00%) violating endpoints...
[ERROR GRT-0226] Type2 ripup not type L.
Error: rsz_timing_postgrt.tcl, 36 GRT-0226
Signal 11 received
Stack trace:
 0# handler(int) in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
 1# _sigtramp in /usr/lib/system/libsystem_platform.dylib
 2# std::__1::__tree<std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>, std::__1::__map_value_compare<sta::Instance*, std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>, std::__1::less<sta::Instance*>, true>, std::__1::allocator<std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>>>::destroy(std::__1::__tree_node<std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>, void*>*) in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
 3# std::__1::__tree<std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>, std::__1::__map_value_compare<sta::Instance*, std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>, std::__1::less<sta::Instance*>, true>, std::__1::allocator<std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>>>::destroy(std::__1::__tree_node<std::__1::__value_type<sta::Instance*, std::__1::tuple<sta::LibertyPort*, sta::LibertyPort*>>, void*>*) in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
 4# rsz::Resizer::~Resizer() in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
 5# rsz::Resizer::~Resizer() in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
 6# ord::OpenRoad::~OpenRoad() in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
 7# __cxa_finalize_ranges in /usr/lib/system/libsystem_c.dylib
 8# exit in /usr/lib/system/libsystem_c.dylib
 9# ord::tclAppInit(Tcl_Interp*) in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
10# Tcl_MainEx in /nix/store/7cpj5zdlz28y6ipg2h4cw6qagvqycm0i-tcl-8.6.13/lib/libtcl8.6.dylib
11# main in /nix/store/x1n09k7mal26ln8z7npr0nhd8qq21846-openroad/bin/.openroad-wrapped
./run_bad.sh: line 17: 37378 Segmentation fault: 11  openroad -exit -no_splash -metrics ./or_metrics_out.json ./scripts/openroad/rsz_timing_postgrt.tcl

Screenshots

No response

Additional Context

No response

maliberty commented 3 months ago

[ERROR GRT-0226] Type2 ripup not type L. suggest the problem is in the fastroute core.

eder-matheus commented 2 months ago

The current incremental grt relies on running grt and rsz in the same script. This is because the internal structures that grt uses to estimate the parasitics and to define what nets are already routed are not present in the database.

To fix this issue, we would need to save the route segments of each net in addition to the route guides. This way rsz will get the correct data about connectivity between the nets.

maliberty commented 2 months ago

If GRT and resizing happens in the same script, this works fine. However, this may be a waste of compute for some designs where GRT takes a non-trivial amount of time.

Most likely any fix is going to involve re-running both. Storing all the extra data in the db seems like a lot of overhead for a small benefit.

The error message should be better regardless.

donn commented 2 months ago

fyi @kareefardi

kareefardi commented 2 months ago

If GRT and resizing happens in the same script, this works fine. However, this may be a waste of compute for some designs where GRT takes a non-trivial amount of time.

Most likely any fix is going to involve re-running both. Storing all the extra data in the db seems like a lot of overhead for a small benefit.

The error message should be better regardless.

Couple of questions:

maliberty commented 2 months ago

You are storing the routing of each net twice - once in guides and again in segments. I don't have a quotative value but that isn't trivial.

I'm not sure what the value of this use model is. Would you explain further how this is a benefit.

donn commented 2 months ago

I'm personally unconcerned. Would you like me to keep this issue open so you can track "The error message should be better regardless"?

maliberty commented 2 months ago

Sure the message should be fixed.

kareefardi commented 2 months ago

@eder-matheus @maliberty I just noticed the commands read_global_route_segments and write_global_route_segments. Would rsz still need an iteration of GRT even with these commands?

maliberty commented 2 months ago

Those were added for a contest and I don't know we'll keep them long term.

eder-matheus commented 1 month ago

@kareefardi This PR will make it possible to estimate parasitics from the guides stored in odb: https://github.com/The-OpenROAD-Project/OpenROAD/pull/5758