The-OpenROAD-Project / OpenROAD

OpenROAD's unified application implementing an RTL-to-GDS Flow. Documentation at https://openroad.readthedocs.io/en/latest/
https://theopenroadproject.org/
BSD 3-Clause "New" or "Revised" License
1.62k stars 562 forks source link

segfault in resizer repair_timing -setup #5889

Open gadfort opened 1 month ago

gadfort commented 1 month ago

Describe the bug

Last night this error was triggered on our CI when updating OpenROAD:

Signal 11 received
Stack trace:
 0# 0x000056071E8EB868 in /sc_tools/bin/openroad
 1# 0x00007F5C92707090 in /lib/x86_64-linux-gnu/libc.so.6
 2# sta::ClkInfo::refsFilter(sta::StaState const*) const in /sc_tools/bin/openroad
 3# sta::TagGroupBldr::setMatchArrival(sta::Tag*, sta::Tag*, float const&, int, sta::PathVertexRep*) in /sc_tools/bin/openroad
 4# sta::ArrivalVisitor::visitFromToPath(sta::Pin const*, sta::Vertex*, sta::RiseFall const*, sta::Tag*, sta::PathVertex*, sta::Edge*, sta::TimingArc*, float, sta::Vertex*, sta::RiseFall const*, sta::Tag*, float&, sta::MinMax const*, sta::PathAnalysisPt const*) in /sc_tools/bin/openroad
 5# sta::PathVisitor::visitFromPath(sta::Pin const*, sta::Vertex*, sta::RiseFall const*, sta::PathVertex*, sta::Edge*, sta::TimingArc*, sta::Pin const*, sta::Vertex*, sta::RiseFall const*, sta::MinMax const*, sta::PathAnalysisPt const*) in /sc_tools/bin/openroad
 6# sta::PathVisitor::visitEdge(sta::Pin const*, sta::Vertex*, sta::Edge*, sta::Pin const*, sta::Vertex*) in /sc_tools/bin/openroad
 7# 0x000056071EC71C54 in /sc_tools/bin/openroad
 8# sta::ArrivalVisitor::visit(sta::Vertex*) in /sc_tools/bin/openroad
 9# 0x000056071EC37A3D in /sc_tools/bin/openroad
10# sta::DispatchQueue::dispatch_thread_handler(unsigned long) in /sc_tools/bin/openroad
11# 0x00007F5C92B10793 in /lib/x86_64-linux-gnu/libstdc++.so.6
12# 0x00007F5C96A61609 in /lib/x86_64-linux-gnu/libpthread.so.0
13# clone in /lib/x86_64-linux-gnu/libc.so.6

The error occurred in repair_timing -setup. This is likely due to some thread safety issues.

Expected Behavior

No segfault

Environment

OpenROAD v2.0-16267-g4ee14b488 
Features included (+) or not (-): +Charts +GPU +GUI +Python

To Reproduce

sc_issue_jpeg_job0_freepdk45_nangate45_cts0_20241007-012707.tar.gz

tar xvf sc_issue_jpeg_job0_freepdk45_nangate45_cts0_20241007-012707.tar.gz
cd sc_issue_jpeg_job0_freepdk45_nangate45_cts0_20241007-012707
./run.sh

This issue appears intermittent, so when I ran it again it went away.

Relevant log output

[INFO CTS-0098] Clock net "clk"
[INFO CTS-0099]  Sinks 4521
[INFO CTS-0100]  Leaf buffers 153
[INFO CTS-0101]  Average sink wire length 846.21 um
[INFO CTS-0102]  Path depth 4 - 5
[INFO CTS-0207]  Leaf load cells 141
[INFO RSZ-0058] Using max wire length 640um.
Placement Analysis
---------------------------------
total displacement        537.5 u
average displacement        0.0 u
max displacement            3.6 u
original HPWL          494447.6 u
legalized HPWL         505048.0 u
delta HPWL                    2 %

Signal 11 received
Stack trace:
 0# 0x000056071E8EB868 in /sc_tools/bin/openroad
 1# 0x00007F5C92707090 in /lib/x86_64-linux-gnu/libc.so.6
 2# sta::ClkInfo::refsFilter(sta::StaState const*) const in /sc_tools/bin/openroad
 3# sta::TagGroupBldr::setMatchArrival(sta::Tag*, sta::Tag*, float const&, int, sta::PathVertexRep*) in /sc_tools/bin/openroad
 4# sta::ArrivalVisitor::visitFromToPath(sta::Pin const*, sta::Vertex*, sta::RiseFall const*, sta::Tag*, sta::PathVertex*, sta::Edge*, sta::TimingArc*, float, sta::Vertex*, sta::RiseFall const*, sta::Tag*, float&, sta::MinMax const*, sta::PathAnalysisPt const*) in /sc_tools/bin/openroad
 5# sta::PathVisitor::visitFromPath(sta::Pin const*, sta::Vertex*, sta::RiseFall const*, sta::PathVertex*, sta::Edge*, sta::TimingArc*, sta::Pin const*, sta::Vertex*, sta::RiseFall const*, sta::MinMax const*, sta::PathAnalysisPt const*) in /sc_tools/bin/openroad
 6# sta::PathVisitor::visitEdge(sta::Pin const*, sta::Vertex*, sta::Edge*, sta::Pin const*, sta::Vertex*) in /sc_tools/bin/openroad
 7# 0x000056071EC71C54 in /sc_tools/bin/openroad
 8# sta::ArrivalVisitor::visit(sta::Vertex*) in /sc_tools/bin/openroad
 9# 0x000056071EC37A3D in /sc_tools/bin/openroad
10# sta::DispatchQueue::dispatch_thread_handler(unsigned long) in /sc_tools/bin/openroad
11# 0x00007F5C92B10793 in /lib/x86_64-linux-gnu/libstdc++.so.6
12# 0x00007F5C96A61609 in /lib/x86_64-linux-gnu/libpthread.so.0
13# clone in /lib/x86_64-linux-gnu/libc.so.6

Screenshots

No response

Additional Context

No response

maliberty commented 1 month ago

do you have the commit ids from the last passing to the first failing?

gadfort commented 1 month ago

81b5450638aeb466300435c6d9caf365aa7ef863 -> 4ee14b488230e046c2e52d9094df85ccf299ac80 Since it's intermittent when I run it on (4ee14b488230e046c2e52d9094df85ccf299ac80) it's not clear that the issue is specific to these commits.

maliberty commented 1 month ago

Try TSAN and ASAN

arthurjolo commented 1 month ago

I tried multiple times to reproduce the crash but wasn't able to. I am going to check TSAN and ASAN for any thread issues.