[BOLT]: Assertion `StateStack.empty() && "corrupt CFI stack"' failed

[llongint]

I encountered this error when I tried to process MySQL sampling file on Arm. Does anyone know how to solve it?

../llvm-project/install/bin/perf2bolt -p=local.data ./install/bin/mysqld -o local.fdata -nl -ignore-build-id

error message:

BOLT-INFO: shared object or position-independent executable detected
PERF2BOLT: Starting data aggregation job for local.pgo.data
PERF2BOLT: spawning perf job to read events without LBR
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: aarch64
BOLT-INFO: BOLT version: 298b86333cb21d8df2d0d8bc62634ecd84690376
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x3920000, offset 0x3920000
BOLT-INFO: enabling relocation mode
BOLT-INFO: disabling -align-macro-fusion on non-x86 platform
BOLT-INFO: enabling strict relocation mode for aggregation purposes
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-WARNING: build-id will not be checked because we could not read one from input binary
PERF2BOLT: waiting for perf mmap events collection to finish...
PERF2BOLT: parsing perf-script mmap events output
FileName -> mmap info:
  ...
  mysqld : 4487 [0xaaaad5482000, 14d5000 @ 1a02000]
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 1 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parsing basic events (without LBR)...
perf2bolt: /home/hezuoqiang/llvm-project/bolt/lib/Core/BinaryFunction.cpp:2374: void llvm::bolt::BinaryFunction::annotateCFIState(): Assertion `StateStack.empty() && "corrupt CFI stack"' failed.
 #0 0x0000aaaad9ac1f90 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (../llvm-project/install3/bin/perf2bolt+0x5c1f90)
 #1 0x0000aaaad9ac063c llvm::sys::RunSignalHandlers() (../llvm-project/install3/bin/perf2bolt+0x5c063c)
 #2 0x0000aaaad9ac24fc SignalHandler(int) Signals.cpp:0:0
 #3 0x0000ffffa6cb48ec (linux-vdso.so.1+0x8ec)
 #4 0x0000ffffa681edf8 (/usr/lib64/libc.so.6+0x80df8)
 #5 0x0000ffffa67dacbc gsignal (/usr/lib64/libc.so.6+0x3ccbc)
 #6 0x0000ffffa67c8cec abort (/usr/lib64/libc.so.6+0x2acec)
 #7 0x0000ffffa67d45d8 (/usr/lib64/libc.so.6+0x365d8)
 #8 0x0000ffffa67d463c (/usr/lib64/libc.so.6+0x3663c)
 #9 0x0000aaaada623bb4 llvm::bolt::BinaryFunction::propagateGnuArgsSizeInfo(unsigned short) (../llvm-project/install3/bin/perf2bolt+0x1123bb4)
#10 0x0000aaaada622b80 llvm::bolt::BinaryFunction::buildCFG(unsigned short) (../llvm-project/install3/bin/perf2bolt+0x1122b80)
#11 0x0000aaaad9b50860 std::_Function_handler<void (llvm::bolt::BinaryFunction&, unsigned short), llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_18>::_M_invoke(std::_Any_data const&, llvm::bolt::BinaryFunction&, unsigned short&&) RewriteInstance.cpp:0:0
#12 0x0000aaaada65d540 llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_2::operator()(std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned short) const ParallelUtilities.cpp:0:0
#13 0x0000aaaad9b601f4 std::_Function_handler<void (), llvm::ThreadPool::createTaskAndFuture(std::function<void ()>)::'lambda'()>::_M_invoke(std::_Any_data const&) (../llvm-project/install3/bin/perf2bolt+0x6601f4)
#14 0x0000aaaada8f2e54 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (../llvm-project/install3/bin/perf2bolt+0x13f2e54)
#15 0x0000aaaada8f391c void* llvm::thread::ThreadProxy<std::tuple<llvm::ThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#16 0x0000ffffa681d320 (/usr/lib64/libc.so.6+0x7f320)
#17 0x0000ffffa688345c (/usr/lib64/libc.so.6+0xe545c)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.

my OS:

# cat /etc/os-release
NAME="openEuler"
VERSION="22.03 (LTS-SP1)"
ID="openEuler"
VERSION_ID="22.03"
PRETTY_NAME="openEuler 22.03 (LTS-SP1)"
ANSI_COLOR="0;31"

[llvmbot]

@llvm/issue-subscribers-bolt

[linsinan1995]

AArch64 might generate mismatched remember/restore CFIs in some cases which could lead to this assertion failure.

related patches: https://reviews.llvm.org/D142975 https://reviews.llvm.org/D147029

[llongint]

AArch64 might generate mismatched remember/restore CFIs in some cases which could lead to this assertion failure.

related patches: https://reviews.llvm.org/D142975 https://reviews.llvm.org/D147029

Thank you very much for your reply. Hope your fixs can be incorporated as soon as possible.

[llongint]

@linsinan1995 This modification does avoid the perf2bolt problem. However, a segment error occurs during running after the llvm-bolt optimization. Is the modification incomplete or is the bug related to the OS configuration?

# numactl -C 77-127 -i 0-3  ./mysqld.opt --defaults-file=/etc/my.cnf --bind-address=127.0.0.1
Segmentation fault (core dumped)

[linsinan1995]

Segmentation fault (core dumped)

it is a bit hard to help you with the limited information in this thread. Can you provide a way to reproduce or more information like llvm commit hash for example?

[llongint]

@linsinan1995 After the -no-pie option is added during compilatin, and the D147029 patch is installed can avoid this problem.

[EugeneZelenko]

@llongint: Patch was not merged yet.