Chapel compiler crashes with LLVM UNREACHABLE executed when called with -O -g and CHPL_LLVM_DEVELOPER=true

milthorpe commented 2 years ago

Summary of Problem

The Chapel compiler crashes with an LLVM error when invoked with -O -g. Neither flag causes a problem if used by itself, nor does --fast -g.

Steps to Reproduce

When I invoke the Chapel compiler on any code e.g. examples/hello.chpl with the flags -O -g, it crashes with the following error:

Remaining virtual register operands
UNREACHABLE executed at /noback/milthorpe/chapel-1.27.0/third-party/llvm/llvm-src/lib/CodeGen/MachineRegisterInfo.cpp:208!
Aborted (core dumped)

Compile command:

chpl -O -g $CHPL_HOME/examples/hello.chpl

Configuration Information

Output of chpl --version:

chpl version 1.27.0
built with LLVM version 14.0.0
Copyright 2020-2022 Hewlett Packard Enterprise Development LP
Copyright 2004-2019 Cray Inc.
(See LICENSE file for more details)

Output of $CHPL_HOME/util/printchplenv --anonymize:

CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: llvm
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native
CHPL_LOCALE_MODEL: flat
CHPL_COMM: none
CHPL_TASKS: qthreads
CHPL_LAUNCHER: none
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
CHPL_GMP: bundled
CHPL_HWLOC: bundled
CHPL_RE2: bundled
CHPL_LLVM: bundled
CHPL_AUX_FILESYS: none

Intel x86 architecture (Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz)
Ubuntu 20.04

milthorpe commented 2 years ago

Chapel 1.26 also crashes for me, with a different error:

chpl: /noback/milthorpe/chapel-1.26.0/third-party/llvm/llvm-src/lib/CodeGen/MachineFunction.cpp:398: void llvm::MachineFunction::DeleteMachineInstr(llvm::MachineInstr*): Assertion `(!MI->isCandidateForCallSiteEntry() || CallSitesInfo.find(MI) == CallSitesInfo.end()) && "Call site info was not updated!"' failed.
Aborted (core dumped)

Output of chpl --version:

chpl version 1.26.0
built with LLVM version 13.0.0
Copyright 2020-2022 Hewlett Packard Enterprise Development LP
Copyright 2004-2019 Cray Inc.
(See LICENSE file for more details)

daviditen commented 2 years ago

@milthorpe Thanks for the report. I was able to reproduce the error locally.

Is this blocking your progress, or are you just reporting the error as a good citizen?

milthorpe commented 2 years ago

The inability to generate debug symbols for either -O or --fast is a blocker, as I can't accurately profile code e.g. using hpctoolkit. (Profiling unoptimized code isn't particularly useful.)

daviditen commented 2 years ago

Are you setting the CHPL_DEVELOPER environment variable? I don't think that assertion should trip without it.

Neither flag causes a problem if used by itself, nor does --fast -g.

Using --fast instead of -O is a workaround you can use then, right?

bradcray commented 2 years ago

@daviditen : I happened to notice you opened https://github.com/chapel-lang/chapel/pull/20237. Does that help with this situation?

daviditen commented 2 years ago

With #20237 compiles work with -O -g and with CHPL_DEVELOPER set. It will still trip the UNREACHABLE assertion if CHPL_LLVM_DEVELOPER is set at LLVM build time though.

bradcray commented 2 years ago

My guess/hope is that merging that will address this issue from @milthorpe's perspective (unless he's wanting/needing to to set CHPL_LLVM_DEVELOPER for other reasons). If true, I'd suggest we remove the 'user issue' label from this since it's only likely to impact developers (even though it was reported by a user). And in either case, @daviditen , would you update the title to reflect the narrower case now?

milthorpe commented 2 years ago

Yes, this fixes the original issue I observed. I'll keep this in mind if I need to set CHPL_LLVM_DEVELOPER. Thanks @daviditen!

chapel-lang / chapel