[Bug]: compiler crash with Release+Asserts build of LLVM and GCC-7.5.0

Summary of Problem

Description:

When I build Chapel with CHPL_LLVM=system I see an assertion from LLVM when compiling even a simple test:

$ chpl ${CHAPEL_SOURCE}/test/release/examples/hello6-taskpar-dist.chpl
chpl: /gpfs/jlse-fs0/users/yeluo/opt/llvm-clang/llvm-project/llvm/include/llvm/IR/Type.h:103: void llvm::Type::setSubclassData(unsigned int): Assertion `getSubclassData() == val && "Subclass data too large for field"' failed.
error: invoking driver compilation phase

The installs of LLVM on this system appear to have been build with assertions enabled, which differs from every other site where I've used CHPL_LLVM=system, as well as differing from how Chapel builds LLVM for CHPL_LLVM=bundled.

Is this issue currently blocking your progress?

No.
I am currently using CHPL_LLVM=bundled on this system, at the expense of additional time spent rebuilding Chapel.

Steps to Reproduce

Source Code:

Using hello6-taskpar-dist.chpl in the Chapel git repo.

Compile command:

$ chpl ${CHAPEL_SOURCE}/test/release/examples/hello6-taskpar-dist.chpl

Execution command:

N/A - compile step failed

Configuration Information

$ chpl --version
chpl version 2.1.0 pre-release (d5a4aa23aa)
  built with LLVM version 17.0.3
  available LLVM targets: xcore, x86-64, x86, wasm64, wasm32, ve, systemz, sparcel, sparcv9, sparc, riscv64, riscv32, ppc64le, ppc64, ppc32le, ppc32, nvptx64, nvptx, msp430, mips64el, mips64, mipsel, mips, loongarch64, loongarch32, lanai, hexagon, bpfeb, bpfel, bpf, avr, thumbeb, thumb, armeb, arm, amdgcn, r600, aarch64_32, aarch64_be, aarch64, arm64_32, arm64
Copyright 2020-2024 Hewlett Packard Enterprise Development LP
Copyright 2004-2019 Cray Inc.
(See LICENSE file for more details)

Note that commit d5a4aa23aa was the current tip of main when I started the build to reproduce this issue.

$ printchplenv --anonymize
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: llvm
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: unknown
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet *
  CHPL_COMM_SUBSTRATE: ibv *
  CHPL_GASNET_SEGMENT: fast +
CHPL_TASKS: qthreads
CHPL_LAUNCHER: gasnetrun_ibv
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
  CHPL_NETWORK_ATOMICS: none
CHPL_GMP: none
CHPL_HWLOC: bundled
CHPL_RE2: none
CHPL_LLVM: system +
CHPL_AUX_FILESYS: none

Also occurs for CHPL_GASNET_SEGMENT=large (not that I'd have expected that to be a factor)

$ /soft/compilers/llvm/release-17.0.3/bin/clang --version
clang version 17.0.3 (https://github.com/llvm/llvm-project.git 888437e1b60011b8a375dd30928ec925b448da57)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /soft/compilers/llvm/release-17.0.3/bin

$ llvm-config --assertion-mode --build-mode --version
ON
Release
17.0.3

Note, in particular, the ON for --assertion-mode.
Also occurs with LLVM 18.1.0 install on the same system (also Release+Asserts)

Other

In a private Slack discussion, @jabraham17 and I agree this looks consistent with Chapel misusing some LLVM API.
However, it could just as well indicate an LLVM bug.

Independent of resolving this particular issue, I want to offer the following change for consideration:

--- a/third-party/llvm/Makefile
+++ b/third-party/llvm/Makefile
@@ -7,7 +7,7 @@ include $(CHPL_MAKE_HOME)/make/Makefile.base

 # decide whether to do a debug or no-debug build
 ifdef CHPL_LLVM_DEVELOPER
-  CHPL_LLVM_DEBUG := -DCMAKE_BUILD_TYPE=Debug -DLLVM_OPTIMIZED_TABLEGEN=1
+  CHPL_LLVM_DEBUG := -DCMAKE_BUILD_TYPE=Debug+Asserts -DLLVM_OPTIMIZED_TABLEGEN=1
 else
   CHPL_LLVM_DEBUG := -DCMAKE_BUILD_TYPE=Release
 endif

chapel-lang / chapel