Installing llvm on macOS Sonoma 14.2.1 with M3 processor fails

[jordydickinson]

See here.

[alan-j-hu]

Looks like LLVM 16 added a dependency on zstd: https://www.phoronix.com/news/LLVM-16.0-Released

I get

$ llvm-config-16 --system-libs --link-static
-lrt -ldl -lm -lz -lzstd -ltinfo -lxml2
$ llvm-config-15 --system-libs --link-static
-lrt -ldl -lm -lz -ltinfo -lxml2

On opam, the llvm package depends on conf-llvm, which should take care of the dependency on LLVM system packages. conf-llvm has a system dependency on zstd for Homebrew but not MacPorts: https://github.com/ocaml/opam-repository/blob/91155d8f7ba41238a91fbffd093e5bb7a532aa0a/packages/conf-llvm/conf-llvm.16/opam#L14. I am not familiar with the Mac ecosystem. So I assume you are capable of installing the conf-llvm opam package despite not having zstd? Are you using MacPorts?

[alan-j-hu]

Ah, I see you are using Homebrew and zstd is installed. I don't have a Mac, so I will likely need your help to troubleshoot this...

[alan-j-hu]

Okay, I was able to reproduce this on Linux. When I try to compile the following test program:

#include <llvm-c/Core.h>

int main()
{
  LLVMContextRef ctx = LLVMGetGlobalContext();
  LLVMModuleRef m = LLVMModuleCreateWithNameInContext("mod", ctx);
  LLVMDumpModule(m);
  LLVMDisposeModule(m);
  return 0;
}

The following works:

$ cc test.c `llvm-config-16 --cflags --system-libs --ldflags --libs Core` -lstdc++
$ ./a.out
; ModuleID = 'mod'
source_filename = "mod"

but the following does not:

$ cc test.c `llvm-config-16 --cflags --system-libs --ldflags --link-static --libs Core` -lstdc++
/usr/bin/ld: cannot find -lzstd: No such file or directory
collect2: error: ld returned 1 exit status

[jordydickinson]

On my system, it appears that the OPAM package finds zstd, as the first error message in my original bug report no longer appears. Instead I get the second output in the original report, which has no mention of zstd but instead seems to be related to the OCaml compiler emitting a warning for missing keyword arguments, and this warning being promoted to an error.

So I think there are two issues going on here. First, the issue with zstd. Note that I did not manually install zstd, so it would have either have had to have been installed (which seems unlikely, as I just got this computer a few days ago), or it was installed by the LLVM OPAM package or some transitive dependency thereof. The second issue is the warnings being promoted to errors that I just mentioned. This seems to suggest that the package is no longer having difficulty finding zstd however.

Both are bizarre. The package fails to install because it cannot find zstd but somehow manages to install it anyway and doesn't fail with this same error on a second attempt. Then the warnings being promoted to errors, which should not happen if the package is being compiled in release mode.

[jordydickinson]

One final note. I previously installed this same package on an older, Intel-based iMac running macOS Catalina or Big Sur (I can't remember exactly). I had no issues with it then, and from what I can tell neither the OPAM package nor LLVM has changed in that time. I believe I installed it on that computer less than three months ago.

[alan-j-hu]

Can you please try to compile the C program I posted using:

cc test.c `llvm-config-16 --cflags --system-libs --ldflags --link-static --libs Core` -lstdc++

and tell me the result? Using the same flags that are used to compile the OCaml package, I am getting an issue compiling the test program on Linux purely using the C API (without any OCaml), so I suspect that the OCaml package is also passing the wrong flags somewhere. Then, I am worried that some broken state in the installation may somehow cause cascading errors, leading to the second error.

When I compile the OCaml bindings from the LLVM source tree directly, I do also get those warnings, but nothing should have been promoted to a hard error.

EDIT: Would it be possible for you to tell me the result of compiling the C program on both macOS Sonoma and macOS Catalina or Big Sur? I am curious if merely compiling the C program leads to either a success or an error on different macOS versions.

[jordydickinson]

I had to make a minor modification to the commands you gave me because the Homebrew version of LLVM is keg-only as follows:

cc test.c `/opt/homebrew/opt/llvm/bin/llvm-config --cflags --system-libs --ldflags --libs Core` -lstdc++ 
cc test.c `/opt/homebrew/opt/llvm/bin/llvm-config --cflags --system-libs --ldflags --link-static --libs Core` -lstdc++

The first succeeds but the second does not. It gives this output:

ld: library 'zstd' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Unfortunately I cannot test this on my older computer as I no longer have it.

[jordydickinson]

It appears pkg-config is able to find it under the name libzstd.

[alan-j-hu]

Okay, thank you for giving the output. I ran ld -lzstd --verbose and got the output:

<likely irrelevant content>
==================================================
ld: mode elf_x86_64
attempt to open /usr/local/lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /usr/local/lib/x86_64-linux-gnu/libzstd.a failed
attempt to open /lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /lib/x86_64-linux-gnu/libzstd.a failed
attempt to open /usr/lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /usr/lib/x86_64-linux-gnu/libzstd.a failed
attempt to open /usr/lib/x86_64-linux-gnu64/libzstd.so failed
attempt to open /usr/lib/x86_64-linux-gnu64/libzstd.a failed
attempt to open /usr/local/lib64/libzstd.so failed
attempt to open /usr/local/lib64/libzstd.a failed
attempt to open /lib64/libzstd.so failed
attempt to open /lib64/libzstd.a failed
attempt to open /usr/lib64/libzstd.so failed
attempt to open /usr/lib64/libzstd.a failed
attempt to open /usr/local/lib/libzstd.so failed
attempt to open /usr/local/lib/libzstd.a failed
attempt to open /lib/libzstd.so failed
attempt to open /lib/libzstd.a failed
attempt to open /usr/lib/libzstd.so failed
attempt to open /usr/lib/libzstd.a failed
attempt to open /usr/x86_64-linux-gnu/lib64/libzstd.so failed
attempt to open /usr/x86_64-linux-gnu/lib64/libzstd.a failed
attempt to open /usr/x86_64-linux-gnu/lib/libzstd.so failed
attempt to open /usr/x86_64-linux-gnu/lib/libzstd.a failed
ld: cannot find -lzstd: No such file or directory
attempt to open /usr/local/lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /usr/local/lib/x86_64-linux-gnu/zstd.a failed
attempt to open /lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /lib/x86_64-linux-gnu/zstd.a failed
attempt to open /usr/lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /usr/lib/x86_64-linux-gnu/zstd.a failed
attempt to open /usr/lib/x86_64-linux-gnu64/libzstd.so failed
attempt to open /usr/lib/x86_64-linux-gnu64/zstd.a failed
attempt to open /usr/local/lib64/libzstd.so failed
attempt to open /usr/local/lib64/zstd.a failed
attempt to open /lib64/libzstd.so failed
attempt to open /lib64/zstd.a failed
attempt to open /usr/lib64/libzstd.so failed
attempt to open /usr/lib64/zstd.a failed
attempt to open /usr/local/lib/libzstd.so failed
attempt to open /usr/local/lib/zstd.a failed
attempt to open /lib/libzstd.so failed
attempt to open /lib/zstd.a failed
attempt to open /usr/lib/libzstd.so failed
attempt to open /usr/lib/zstd.a failed
attempt to open /usr/x86_64-linux-gnu/lib64/libzstd.so failed
attempt to open /usr/x86_64-linux-gnu/lib64/zstd.a failed
attempt to open /usr/x86_64-linux-gnu/lib/libzstd.so failed
attempt to open /usr/x86_64-linux-gnu/lib/zstd.a failed

I checked my package manager and I have the libzstd1 package installed, which provides the following files:

/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libzstd.so.1
/usr/lib/x86_64-linux-gnu/libzstd.so.1.4.8
/usr/share
/usr/share/doc
/usr/share/doc/libzstd1
/usr/share/doc/libzstd1/changelog.Debian.gz
/usr/share/doc/libzstd1/copyright

Then, I installed libzstd-dev. After I did so, the compilation succeeded and the program ran correctly. ld -lzstd --verbose gives:

==================================================
ld: mode elf_x86_64
attempt to open /usr/local/lib/x86_64-linux-gnu/libzstd.so failed
attempt to open /usr/local/lib/x86_64-linux-gnu/libzstd.a failed
attempt to open /lib/x86_64-linux-gnu/libzstd.so succeeded
/lib/x86_64-linux-gnu/libzstd.so
libc.so.6 needed by /lib/x86_64-linux-gnu/libzstd.so
<output about locating libc>

Looks like adding a dependency on libzstd-dev for Ubuntu-based Linuxes should fix the problem on my end. Do you know what modifications would be needed to fix this problem for Homebrew?

[alan-j-hu]

Actually, the conf-llvm package already specifies a system dependency on libzstd-dev for Ubuntu, but I didn't have it installed at the moment. Can you post the result of ld -lzstd --verbose and what zstd-related files you have installed?

[jordydickinson]

The command ld -lzstd --verbose fails:

ld: library not found for -lzstd

and macOS system ld apparently doesn't recognize --verbose anyway. But using LLVM's lld and adding a -L flag (obtained from pkg-config), I get the following output (command is /opt/homebrew/opt/llvm/bin/ld.lld -L/opt/homebrew/opt/zstd/lib -lzstd --verbose):

ld.lld: /opt/homebrew/opt/zstd/lib/libzstd.a
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'debug.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'entropy_common.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'error_private.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'fse_decompress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'pool.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'threading.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'xxhash.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_common.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'fse_compress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'hist.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'huf_compress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_compress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_compress_literals.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_compress_sequences.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_compress_superblock.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_double_fast.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_fast.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_lazy.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_ldm.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_opt.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstdmt_compress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'huf_decompress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'huf_decompress_amd64.S.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_ddict.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_decompress.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_decompress_block.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'cover.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'divsufsort.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'fastcover.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zdict.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v01.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v02.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v03.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v04.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v05.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v06.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: warning: /opt/homebrew/opt/zstd/lib/libzstd.a: archive member 'zstd_v07.c.o' is neither ET_REL nor LLVM bitcode
ld.lld: error: target emulation unknown: -m or at least one .o file required

[jordydickinson]

Interestingly, Homebrew doesn't indicate that zstd is keg-only, but it doesn't add a symlink for it in /usr/local (as it normally would for packages that aren't keg-only).

[alan-j-hu]

Thank you for bearing with me. I'm not a Mac person and I realize that I haven't been very clear in all my posts on this issue tracker; I'm just poking around and trying to troubleshoot despite not having access to a Mac. I'm trying to think if this is an issue with the llvm opam package, the conf-llvm opam package, a Homebrew package, and basically who is "responsible" for fixing it. I'm reluctant to introduce "hacky" fixes into the LLVM package if the issue is something downstream, as I don't have access to a Mac and won't be able to test things easily.

Do you think this is an issue with the zstd Homebrew package and that it needs to make a symlink? Is https://stackoverflow.com/questions/67840691/ld-library-not-found-for-lzstd-while-bundle-install-for-mysql2-gem-ruby-on-mac related?

[alan-j-hu]

Also, the LLVM 16 opam package was published on October 11, less than three months ago, while the LLVM 15 opam package was published on September 4. LLVM 16 added a dependency on zstd. You state that the LLVM installation was working "less than three months ago." I don't know how the dates line up with you; is it possible that you had LLVM 15 installed before (which did not depend on zstd), so the installation worked, but now you can't install LLVM 16 due to issues related to how zstd is packaged on Homebrew?

[jordydickinson]

So it appears I was wrong w.r.t. the symlinking: Homebrew on Apple silicon (Apple's M-series ARM processors) uses /opt/homebrew as the installation prefix rather than /usr/local (as it used to do for Intel-based Macs). So I don't think it's an error on Homebrew's end as zstd adds the appropriate symlink at /opt/homebrew/lib/libzstd.a.

It is interesting though that the problem you linked to also occurs on an Apple silicon iMac. Could the package be trying to find zstd in /usr/local?

I don't know how the dates line up with you; is it possible that you had LLVM 15 installed before (which did not depend on zstd), so the installation worked, but now you can't install LLVM 16 due to issues related to how zstd is packaged on Homebrew?

It is not possible that I had been using LLVM 15. I'm certain I had LLVM 16. So I must have installed it at some point after October 11th.

[jordydickinson]

It is interesting though that the problem you linked to also occurs on an Apple silicon iMac. Could the package be trying to find zstd in /usr/local?

According to man ld, macOS's ld searches /usr/lib then /usr/local/lib. Adding the flag -L/opt/homebrew/lib seems to allow ld to find libzstd.a.

[jordydickinson]

Earlier I stated that opam install llvm no longer reports an error involving zstd. Looking at the output file in the error message indicates that it is in fact still failing to find zstd.

[alan-j-hu]

I also found this issue: https://discourse.llvm.org/t/kaleidoscope-on-m1-mac-help/69010

My current suspicion is that Mac (and perhaps this has to do with the new M1 Macs) broke something related to library search paths that is causing issues when linking to zstd. I am interested in what the search path is when ld looks for zstd.

[jordydickinson]

My current suspicion is that Mac (and perhaps this has to do with the new M1 Macs) broke something related to library search paths that is causing issues when linking to zstd. I am interested in what the search path is when ld looks for zstd.

From what I've been reading (see here), Homebrew switched from /usr/local to /opt/homebrew so that older Intel binaries (which run under Rosetta2) installed by Intel-based Homebrew installations could live in /usr/local. Unfortunately, macOS has no way to set a global library path, and variables like LD_LIBRARY_PATH are ignored by child processes as a security feature. So Homebrew has no way to fix the library path until and unless Apple either hardcodes the new Homebrew paths or adds a feature to customize the paths.

So in summary, yes, that is exactly what appears to have happened, but Homebrew can't fix it.

It's worth noting that pkg-config provides the correct flags for libzstd. Is there any way the package can use these flags instead?

[jordydickinson]

Given the installed llvm-config doesn't add the correct path to link with zstd when --link-static is provided, I'd say the problem lies with either LLVM itself or the Homebrew LLVM package's configuration.

[alan-j-hu]

I'm reading about this Homebrew change and thinking about the best solution. I found a big discussion about this change over at https://github.com/Homebrew/brew/issues/9177. The issue states:

Today, Homebrew's prefix is /usr/local, with the repository in /usr/local/Homebrew. In the initial phase of Apple Silicon migration, in order to avoid using the same prefix for both Intel and ARM binaries, we have two prefixes instead: /usr/local for Intel, and /opt/homebrew for Apple Silicon. Eventually, the need for /usr/local as a Rosetta installation of Homebrew will go away and Apple Silicon users will have only one installation.

Way back in the mists of history, /usr/local was chosen for a few reasons:

1. /usr/local/bin is in the default PATH, so most things a user will use will end up seeing Homebrew-installed tools without configuration.
2. Buildsystems look in /usr/local/include, /usr/local/lib and /usr/local/lib/pkgconfig by default, so non-Homebrew software will find Homebrew-installed libraries without configuration.

These reasons were all in contrast to MacPorts, which installed its software in /opt/local. /usr/local was chosen in response to the issues users were experiencing at the time. We have also received some feedback from users who would prefer a different default prefix, namely:

1. Some non-Homebrew tools also install libraries and binaries in /usr/local, for the same reason Homebrew does, and so there are sometimes conflicts between them.
2. Some users don't want Homebrew-installed software to be found by default / without configuration, making an alternate prefix preferable.

So, the primary reason for the change is so that Intel-based binaries and ARM-based binaries can coexist during migration to Apple Silicon. A secondary reason for the change is that /usr/local is the "default," and people did not want Homebrew-installed packages to be findable by default.

fxcourdert says: https://github.com/Homebrew/brew/issues/9177#issuecomment-732795544

I think /opt/homebrew is the least painful of choices, so let's go with that. We'll need to document it clearly, and it will break a ton of stuff (in particular, a lot of Python packages look for libraries in /usr/local/lib).

and https://github.com/Homebrew/brew/issues/9177#issuecomment-733553584

configure environment variables (PATH, LIBRARY_PATH, DYLD_LIBRARY_PATH, CMAKE_PREFIX_PATH, PKG_CONFIG_PATH, etc)

That's the main downside of leaving /usr/local: many build scripts and compiler tools look there by default. It was cited as an upside by some, though, so maybe we don't want to mess with variables other than PATH. Also, the number of possible variables to add is endless, as each build system has its own.

(DYLD_LIBRARY_PATH should not be necessary, that's for libraries copied outside of their install path, which brew does fix for us automatically.)

So, I am getting conflicting messages about the intent of this change. It seems that the Homebrew developers specifically don't want other tooling to find Homebrew-installed libraries by default, while acknowledging that it will break other software?

Does Homebrew recommend a specific way to make its installations visible to other software by default?

[jordydickinson]

In the past, Homebrew installed symlinks into /usr/local except for certain packages marked as "keg-only", primarily for software that conflicted with macOS system software (e.g., LLVM is keg-only because Xcode on macOS installs its own clang).

Eventually, the need for /usr/local as a Rosetta installation of Homebrew will go away and Apple Silicon users will have only one installation.

My reading of this is that they don't want to change the old behavior, and the distinction exists primarily to allow the coexistence of Intel and ARM binaries during the transition to Apple Silicon. Note also that the default prefix is changeable, which addresses the use-case of users that don't want Homebrew packages to be found by default.

So, I am getting conflicting messages about the intent of this change. It seems that the Homebrew developers specifically don't want other tooling to find Homebrew-installed libraries by default, while acknowledging that it will break other software?

My interpretation is that the primary reason for the change was to allow the coexistence of Intel and Apple Silicon binaries. I doubt their primary intention was to make Homebrew-installed software not discoverable by default, as they have both keg-only packages (to eliminate conflicts with macOS system software) and the option to change the default prefix (allowing for users to make Homebrew-installed software not discoverable by default).

I really hope that wasn't their intention, because I don't see the point of a package manager that doesn't put installed binaries in PATH.

Does Homebrew recommend a specific way to make its installations visible to other software by default?

With the exception of keg-only software, all Homebrew software is visible by default. For keg-only packages, Homebrew has a "Caveats" section in the package info telling you how to access it. (For instance, the caveats for LLVM mention the appropriate LDFLAGS, CPPFLAGS, and PATH variables that need to be set.)

[alan-j-hu]

In the past, Homebrew installed symlinks into /usr/local except for certain packages marked as "keg-only", primarily for software that conflicted with macOS system software (e.g., LLVM is keg-only because Xcode on macOS installs its own clang). ... With the exception of keg-only software, all Homebrew software is visible by default. For keg-only packages, Homebrew has a "Caveats" section in the package info telling you how to access it. (For instance, the caveats for LLVM mention the appropriate LDFLAGS, CPPFLAGS, and PATH variables that need to be set.)

Note that the problematic package is zstd, not LLVM. zstd does not have a "Caveats" section. zstd is not a "keg-only" package.

My reading of this is that they don't want to change the old behavior, and the distinction exists primarily to allow the coexistence of Intel and ARM binaries during the transition to Apple Silicon. Note also that the default prefix is changeable, which addresses the use-case of users that don't want Homebrew packages to be found by default. ... My interpretation is that the primary reason for the change was to allow the coexistence of Intel and Apple Silicon binaries. I doubt their primary intention was to make Homebrew-installed software not discoverable by default, as they have both keg-only packages (to eliminate conflicts with macOS system software) and the option to change the default prefix (allowing for users to make Homebrew-installed software not discoverable by default).

I agree that the main reason why Homebrew changed prefix is to allow Intel and ARM binaries to coexist. The reasoning in the Homebrew issue about not wanting Homebrew installations to be visible by default, or interfere with other installations in /usr/local, seems to be weaker ex-post-facto reasoning, a "justification" for a change motivated by the new Mac M1.

If software is installed in a non-standard location, I don't think it's LLVM's responsibility to automatically find it. If I save a zstd shared object file in some random location, why should llvm-config find it? However, opam advertises compatibility with Homebrew, with the os-distribution = "homebrew" filter field for depexts. One should expect that opam works with Homebrew out of the box. It's either Homebrew's responsibility to fix for breaking zstd, a non-keg-only package, and not providing a clear workaround, or opam's responsibility because one of its functionalities is compatibility with Homebrew, so it should adapt to changes to Homebrew's functionality.

I have a fix at https://github.com/alan-j-hu/llvm-dune/tree/llvm-16-homebrew. It adds a new dependency on pkg-config The package continues to work on Linux, and it should fix the issue for ARM-based Macs using Homebrew. Should I merge it and make a new release?

[jordydickinson]

Note that the problematic package is zstd, not LLVM. zstd does not have a "Caveats" section. zstd is not a "keg-only" package.

I disagree with this. zstd is indeed not keg-only, but the problem is not with the zstd package but with Homebrew and macOS. Before Apple Silicon, zstd would be found in standard search locations in /usr/local, but the addition of /opt/homebrew breaks this, and Homebrew itself cannot add additional search paths to macOS due to macOS's System Integrity Protection. This is a security feature that (among other things) removes the relevant environment variables before spawning a child process, making the environment variables effectively useless. From the discussion here, it appears that Homebrew is aware of the issue, unable to fix it, and blames Apple. (I agree with them on this.)

If software is installed in a non-standard location, I don't think it's LLVM's responsibility to automatically find it. If I save a zstd shared object file in some random location, why should llvm-config find it?

The purpose of llvm-config is to provide the proper compilation and linking flags, in the same way pkg-config is responsible for the same thing. Adding a -L for zstd is, in my opinion, part of this purpose. I'm not sure it's a problem with LLVM itself. For example, perhaps the Homebrew package is improperly configuring LLVM such that it cannot find zstd. So if llvm-config cannot be configured with the appropriate flags for zstd, then this is a problem with LLVM. If it can and Homebrew is not doing this correctly, then it's a problem with Homebrew.

It's either Homebrew's responsibility to fix for breaking zstd, a non-keg-only package, and not providing a clear workaround, or opam's responsibility because one of its functionalities is compatibility with Homebrew, so it should adapt to changes to Homebrew's functionality.

I don't think the issue is with zstd for the reasons outlined above, nor is it an issue with OPAM. I believe it's an issue with either LLVM's llvm-config or Homebrew's LLVM package.

I have a fix at https://github.com/alan-j-hu/llvm-dune/tree/llvm-16-homebrew. It adds a new dependency on pkg-config The package continues to work on Linux, and it should fix the issue for ARM-based Macs using Homebrew. Should I merge it and make a new release?

I think that's a wonderful idea. I would test it on my machine for you but I don't know how to configure OPAM to use the release. As stated earlier, however, I think the real issue is with either llvm-config or Homebrew's LLVM package. Ultimately Homebrew itself has an issue which is currently unfixable as it requires either cooperation with Apple to add new default search paths or for the transition to Apple Silicon to be complete, at which point /opt/homebrew can replace /usr/local. While I don't think the issue with Homebrew to be the root cause of this problem, fixing Homebrew would also eliminate this problem.

[jordydickinson]

I managed to figure out how to add the updated package to OPAM, but I get this error:

[ERROR] Package conflict!
  * Missing dependency:
    - conf-llvm >= 16.0.6+nnp
    no matching version

[jordydickinson]

According to this comment, Homebrew believes this to be a problem with llvm-config. They also indicate that llvm-config has had many issues in the past (esp. on macOS) and will likely be deprecated and replaced with CMake export.

[alan-j-hu]

When I run llvm-config-16 --link-static --system-libs, I get the output -lrt -ldl -lm -lz -lzstd -ltinfo -lxml2. If you look at the source code for llvm-config, you'll see that this command just prints out the contents of a CMake variable, LLVM_SYSTEM_LIBS, defined in BuildVariables.inc.in. There is no dynamic lookup on the user's system for the locations of these libraries. LLVM just assumes they are installed in standard locations.

I looked at the zstd "formula" on Homebrew. Is this line why pkg-config is able to find zstd when Homebrew installs it in a non-standard location?

[jordydickinson]

I looked at the zstd "formula" on Homebrew. Is this line why pkg-config is able to find zstd when Homebrew installs it in a non-standard location?

From what I can tell, yes. pkg-config itself is also installed in a non-standard location but while SIP prevents environment variables for linker search paths to be useful it doesn't prevent updating PATH and Homebrew's version of pkg-config apparently correctly searches /opt/homebrew/lib/pkgconfig, which is where libzstd.pc is located.