GRT crashes for mpl2 change secure-ci run when repairing setup and hold violations

AcKoucher commented 3 days ago

Subject

[Stage]: Global Router.

Describe the bug

Segfault during grt stage when repairing setup and hold violations for sky130hd/riscv32i. Secure-CI run.

Expected Behavior

No crash.

Environment

HEAD of secure-mpl2-flat-vertices

To Reproduce

run flow using the secure-mpl2-flat-vertices branch

Relevant log output

Repair setup and hold violations...
Signal 8 received
Stack trace:
 0# 0x0000000000E07AA7 in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 1# 0x00007FA93965E400 in /lib64/libc.so.6
 2# sta::TagGroup::arrivalIndex(sta::Tag*, int&, bool&) const in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 3# sta::PathVertex::PathVertex(sta::PathVertexRep const&, sta::StaState const*) in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 4# sta::CheckCrpr::maxCrpr(sta::ClkInfo*) in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 5# sta::ArrivalVisitor::pruneCrprArrivals() in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 6# sta::ArrivalVisitor::visit(sta::Vertex*) in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 7# 0x0000000001084209 in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 8# sta::DispatchQueue::dispatch_thread_handler(unsigned long) in /tmp/workspace/_small_secure-mpl2-flat-vertices/tools/install/OpenROAD/bin/openroad
 9# 0x00007FA93D59AEB4 in /home/jenkins/.local/lib64/libortools.so.9
10# 0x00007FA93E0E3EA5 in /lib64/libpthread.so.0
11# clone in /lib64/libc.so.6
Elapsed time: 0:02.60[h:]min:sec. CPU time: user 3.50 sys 1.53 (193%). Peak memory: 248056KB.

Screenshots

No response

Additional Context

No response

maliberty commented 3 days ago

I've seen this intermittently but haven't been able to stable reproduce it or catch anything with valgrind. Can you reproduce it?

AcKoucher commented 3 days ago

I couldn't reproduce it locally

eder-matheus commented 3 days ago

@maliberty My first thought was that jenkins might be killing the process and it shows this stack trace. I will try some runs locally.

maliberty commented 3 days ago

It seems to be a fairly similar stack each time. I've also seen it in GCD so I didn't think it is a timeout

maliberty commented 3 days ago

I've been thinking of trying asan or tsan

The-OpenROAD-Project / OpenROAD-flow-scripts