Building Chapel from source on Homebrew fails if hwloc, jemalloc, or re2 are installed

[bradcray]

When building Chapel from source (brew install --build-from-source chapel) using the homebrew formula (which is now what happens by default if you are a Mojave user since Mojave is no longer supported), the build seems to fail if any of the following homebrew packages are installed:

• hwloc
• jemalloc
• re2

However, building from Chapel from a source release or the main branch does not have this same problem even if those packages are installed. My guess is that in homebrew's build process, homebrew's paths precede Chapel's such that their versions "win" when we're looking for headers or libraries. Uninstalling the homebrew packages in question will fix the build, but other homebrew packages may depend on their existence, so this isn't an ideal solution.

So the main question here is what more we could do with our homebrew formula to avoid these conflicts when building from source?

Here are some of the failure modes in case anyone searches on some of the strings in question:

hwloc:

==> make
Last 15 lines from /Users/bradc/Library/Logs/Homebrew/chapel/07.make:
checking sys/utsname.h usability... yes
checking sys/utsname.h presence... yes
checking for sys/utsname.h... yes
checking for mach_absolute_time... yes
checking for high resolution timer type... mach
checking hwloc.h usability... yes
checking hwloc.h presence... yes
checking for hwloc.h... yes
checking for library containing chpl_hwloc_topology_init... no
configure: error: Specified topology library (hwloc_via_chapel) does not work.
make[4]: *** [qthread-config] Error 1
make[3]: *** [/usr/local/Cellar/chapel/1.25.1/libexec/third-party/qthread/install/darwin-x86_64-native-llvm-none-flat-jemalloc-bundled] Error 2
make[2]: *** [third-party-pkgs] Error 2
make[1]: *** [runtime] Error 2
make: *** [comprt] Error 2

jemalloc:

==> make mason
Last 15 lines from /Users/bradc/Library/Logs/Homebrew/chapel/09.make:
      _qio_writev in libchpl.a(qio.o)
      _qio_preadv in libchpl.a(qio.o)
      _qio_freadv in libchpl.a(qio.o)
      _qio_fwritev in libchpl.a(qio.o)
      _qio_pwritev in libchpl.a(qio.o)
      ...
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
error: Make Binary - Linking
  Error building mason. Check that you have built a runtime configuration that supports:
  CHPL_COMM=none
  CHPL_LAUNCHER=none
  CHPL_RE2=bundled
make[1]: *** [/usr/local/Cellar/chapel/1.25.1/libexec/bin/darwin-x86_64/mason] Error 1
make: *** [mason] Error 2

re2:

==> make mason
Last 15 lines from /Users/bradc/Library/Logs/Homebrew/chapel/09.make:
make[2]: Nothing to be done for `default'.
cd tools/mason && /Library/Developer/CommandLineTools/usr/bin/make && /Library/Developer/CommandLineTools/usr/bin/make install
Building Mason...
Undefined symbols for architecture x86_64:
  "re2::RE2::MatchFile(re2::FilePiece&, re2::StringPiece const&, re2::RE2::Anchor, re2::FilePiece*, int) const", referenced from:
      _qio_regex_channel_match in libchpl.a(re2-interface.o)
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
error: Make Binary - Linking
  Error building mason. Check that you have built a runtime configuration that supports:
  CHPL_COMM=none
  CHPL_LAUNCHER=none
  CHPL_RE2=bundled
make[1]: *** [/usr/local/Cellar/chapel/1.25.1/libexec/bin/darwin-x86_64/mason] Error 1
make: *** [mason] Error 2

[bradcray]

Though I saw this on Mojave, @aconsroe-hpe reproduced one of these failure modes on Big Sur when building from source, so I think it's independent of OS version and only based on building from source.

[mppf]

I believe this is from #18840 and in PR #18880 I am working on solving it. But,

However, building from Chapel from a source release or the main branch does not have this same problem even if those packages are installed.

If that's the case, perhaps it is a different issue from #18840 / #18880. It would be interesting to check a Homebrew build with the formula after that PR.

Anyway, note that the Homebrew formula patched main to use brew --prefix to add a -I path based on it. We were previously adding /usr/local/include to the search path but it's possible that is not happening on your system for one reason or other. AFAIK we should be doing this with brew --prefix and getting /usr/local/include in most Homebrew settings but that would cause the described problem if it is enabled. Until #18840 / #18880.

[bradcray]

Thanks for the notes. I'm not sure whether this strengthens or weakens your suspicions (or has no bearing on them), but I believe that I was seeing this when building using the 1.25.0 formula from source as well. I don't think I had tried building from source prior to that formula.

[my guess: no bearing since the path ordering issue has been since time immemorial?]

[cassella]

I'm seeing the hwloc error today on Linux,

...
checking hwloc.h usability... yes
checking hwloc.h presence... yes
checking for hwloc.h... yes
checking for library containing chpl_hwloc_topology_init... no
configure: error: Specified topology library (hwloc_via_chapel) does not work.
make[4]: *** [Makefile:187: qthread-config] Error 1
make[3]: *** [Makefile:174: /home/fortytwo/src/chapel/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-bundled] Error 2

This persists after a make clean.

DISTRIB_ID=Pop
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Pop!_OS 22.04 LTS"

This is on the current main commit 59543df77

CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: llvm
CHPL_TARGET_ARCH: x86_64
CHPL_TARGET_CPU: native *
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet *
  CHPL_COMM_SUBSTRATE: udp
  CHPL_GASNET_SEGMENT: everything
CHPL_TASKS: qthreads
CHPL_LAUNCHER: amudprun
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
  CHPL_NETWORK_ATOMICS: none
CHPL_GMP: none
CHPL_HWLOC: bundled
CHPL_RE2: bundled
CHPL_LLVM: system
CHPL_AUX_FILESYS: none

[bradcray]

Huh... Paul, is this on a system that I/we would have access to?

[cassella]

Not easily -- it's my laptop.

I found I could build with CHPL_LLVM=none.

I was also able to build without that but with CHPL_LIB_PIC=pic.

I saw 3 attempts in the qthread configure log to find that chpl_hwloc_topology_init symbol:

clang ...  conftest.c -lm  >&5
/usr/bin/ld: /tmp/conftest-184d57.o: in function `main':
conftest.c:(.text+0x12): undefined reference to `chpl_hwloc_topology_init'

clang ...  conftest.c -lhwloc  -lm  >&5
/usr/bin/ld: /home/fortytwo/src/chapel/third-party/hwloc/install/linux64-x86_64-native-llvm-none-flat/lib/libhwloc.a(topology.o): relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: failed to set dynamic section sizes: bad value

and

clang ... conftest.c -lhwloc -lnuma  -lm  >&5
/usr/bin/ld: cannot find -lnuma: No such file or directory

(the second of which is what led me to try CHPL_LIB_PIC.)

The installed system llvm is

Version: 1:14.0-55~exp2

[mppf]

@cassella - it sounds like you are on something like Ubuntu 22.04. Do you have hwloc, jemalloc, or re2 installed? Or is it an unrelated problem to this issue's original post?

[cassella]

@mppf Yes, it's something like ubuntu 22.04. I don't have any of those packages installed. My issue seems more likely now to be unrelated -- I'd just initially seen the same error message and latched onto that rather than opening a new issue.

[cassella]

Well, whatever it was, make clobber fixed it for me.

[mppf]

@bradcray — any idea if the original problem described here is still present with 1.28?

[bradcray]

any idea if the original problem described here is still present with 1.28?

I'm giving it a try...

[bradcray]

I ended up having to kill that build because it was dragging my entire system down. Can try again on some proverbial quiet day.

[stonea]

We've seen this issue again on our nightly build where we've tried to have a Homebrew formula that used our bundled version of jemalloc (and have since modified things to use a system version).

What appears to be happening is that when Homebrew builds the formula it uses its own shim version of ld (see https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/ld which links to https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/cc), which adds in -L paths to existing homebrew installed libraries. You can see this by setting HOMEBREW_CC_LOG_PATH to the name of some log file, building, and looking at the resulting log.

I'm thinking ultimately, it might be better for Chapel to give its third-party deps unique names (e.g. chpl_jemalloc.a instead of jemalloc.a) so there's no possibility of confusion. But this may be complicated by the fact that all these third-party deps have their own build systems, output multiple libraries, etc.

Alternatively, if we always statically link this, maybe we could just add in /absolute/path/to/our/bundled/jemalloc.a directly to the linker command rather than passing it via -l?

[arezaii]

testing this today, I think only the re2 problem still exists (at least that is the only one I could recreate), as we added jemalloc and hwloc as brew deps in 2.1

[bradcray]

I feel like I saw something about system re2 installations in @jabraham17 's recent changes w.r.t. these settings to get homebrew working better, but can't remember what or where… Oh, maybe I'm thinking of this line from the CHANGES.md file:

* added an error message if the user attempts to set `CHPL_RE2=system`

(and maybe I asked Jade a follow-up question about what it was and why at that time?) Anyway, Jade, can you remind us what you know about re2 here?

[jabraham17]

CHPL_RE2=system is not supported, and as of 2.1, explicitly disabled in the build system. However, I did not address the problem of homebrews linker trying to use the homebrew installed re2 and have seen on my own system that having re2 installed breaks the chapel build from source.

The reason we don't support CHPL_RE2=system is because we have patched the bundled re2 with new features that were not accepted into the main branch of re2. So we cannot use regular re2, unless those patches were contributed back to re2. @mppf is the expert on the details here if that is relevant to this discussion

[mppf]

Some of the discussion in https://github.com/chapel-lang/chapel/pull/25405 could be related to failing if re2 is installed. Or, perhaps it's a problem with Homebrew's wrappers requiring us to use the installed re2.