`zstd` directory not found after updating the library

[voodoos]

I encountered a surprising error when working with a custom compiler recently. It looks like the compiler is expecting headers for libzstd to be in a directory that might change when the lib is updated. I don't now if this is the expected behaviour, but it feels wrong to have to re-configure and re-build the compiler after such an update.

Environment:

• Platform: Mac M1
• Compiler version: a branch based on 5.1
• libzstd installed with brew

Steps to reproduce:

• Configure and build the compiler
• Update the zstd library with brew (in my case from 1.5.2 to 1.5.5)
• Link an executable with the compiler

$ echo "print_int 42" > main.ml
$ ocamlopt.opt main.ml
ld: warning: directory not found for option '-L/opt/homebrew/Cellar/zstd/1.5.2/lib'
ld: library not found for -lzstd
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)

Output of ocamlc -config:

bytecomp_c_libraries:  -L/opt/homebrew/Cellar/zstd/1.5.2/lib -lzstd -lm  -lpthread
native_c_libraries:  -L/opt/homebrew/Cellar/zstd/1.5.2/lib -lzstd -lm  -lpthread

Output of otool -L _opam/bin/ocamlopt.opt (otoolis a ldd equivalent)

_opam/bin/ocamlopt.opt:
        /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.2)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

[nojb]

I would say this is expected, as the compilation flags for libzstd are queried at configure-time. In systems where libzstd is stored in a non-standard place (like Homebrew-based systems), those flags will include the path to the library, and this is not resistant to upgrades.

Without making pkg-config a runtime dependency of the compiler (which we surely don't want to do), I don't see how to do better. If someone more knowledgeable about Homebrew knows whether there is a way to write down compilation flags for libzstd which is resistant to upgrades, we could try to integrate this logic into the configure script.

[xavierleroy]

I think this is an oddity in the way Homebrew uses pkg-config. The .pc file for libzstd could very well point to /opt/homebrew/lib as the location of -lzstd, since there's a symbolic link there to the actual location. But they chose to point directly to the version-specific Cellar directory. I suspect OCaml is not the only user of libzstd that can need reconfiguration after a libzstd upgrade.

Maybe that's a question for the Homebrew forums?

[voodoos]

Maybe that's a question for the Homebrew forums?

I opened a discussion on brew's github: https://github.com/orgs/Homebrew/discussions/4682

[osalbahr]

whether there is a way to write down compilation flags for libzstd which is resistant to upgrades

Are you able to run code during that period? If so, you can dynamically set the -L... flag as follows:

$ find "$(brew --cellar zstd)" | grep lib$
/home/linuxbrew/.linuxbrew/Cellar/zstd/1.5.5/lib
$ ls "$(find "$(brew --cellar zstd)" | grep lib$)"
cmake  libzstd.a  libzstd.so  libzstd.so.1  libzstd.so.1.5.5  pkgconfig

Alternatively:

$ cd "$(brew --cellar zstd)"/*/lib
$ pwd
/home/linuxbrew/.linuxbrew/Cellar/zstd/1.5.5/lib

The reason is that Homebrew is a rolling release package manager, meaning there should be only one directory under brew --cellar [formula] even after upgrading. Like so:

$ echo "$(brew --cellar zstd)"
/home/linuxbrew/.linuxbrew/Cellar/zstd
$ ls -a "$(brew --cellar zstd)"
.  ..  1.5.5

Note: the full path of brew --cellar depends on brew --prefix. Homebrew's prefix is a fixed directory depending on the system. For me, that is /home/linuxbrew/.linuxbrew but for OP that's /opt/homebrew.

[osalbahr]

The .pc file for libzstd could very well point to /opt/homebrew/lib as the location of -lzstd, since there's a symbolic link there to the actual location.

Actually, yeah that's right.

$ find "$(brew --prefix)/lib" -name '*zstd*'
/home/linuxbrew/.linuxbrew/lib/pkgconfig/libzstd.pc
/home/linuxbrew/.linuxbrew/lib/cmake/zstd
/home/linuxbrew/.linuxbrew/lib/libzstd.a
/home/linuxbrew/.linuxbrew/lib/libzstd.so
/home/linuxbrew/.linuxbrew/lib/libzstd.so.1
/home/linuxbrew/.linuxbrew/lib/libzstd.so.1.5.5
$ ls -l /home/linuxbrew/.linuxbrew/lib/libzstd.a
lrwxrwxrwx. 1 dracula dracula 34 Jul  9 18:53 /home/linuxbrew/.linuxbrew/lib/libzstd.a -> ../Cellar/zstd/1.5.5/lib/libzstd.a

Did OP set -L... manually or did Homebrew set it somewhere? Not sure I'm following.

@voodoos could you show the output of ./configure and sh -x ./configure?

[osalbahr]

Side question: does brew install ocaml fit your use case or do you need to build it from source?

$ echo "print_int 42" > main.ml
$ ocamlopt.opt main.ml
$ ./a.out 
42$ 
$ which ocamlopt.opt
/home/linuxbrew/.linuxbrew/bin/ocamlopt.opt
$ brew install ocaml
Warning: ocaml 4.14.0 is already installed and up-to-date.
To reinstall 4.14.0, run:
  brew reinstall ocaml

[voodoos]

Thanks for your prompt reply @osalbahr !

Did OP set -L... manually or did Homebrew set it somewhere? Not sure I'm following.

I have not set anything manually. The idea is that users' programs built by the compiler have to be linked against zstd. The path to the lib is detected when the compiler itself is configured. Then, when the compiler is called to build and link some OCaml code, it will pass the path of the zstd lib to the linker. The same path that was detected at configuration time of the compiler.

@voodoos could you show the output of ./configure and sh -x ./configure?

I guess you mean the configure script of the compiler ? Here are the outputs:

./configure

configure: Configuring OCaml version 5.1.0+dev0-2023-04-11
checking build system type... aarch64-apple-darwin22.4.0
checking host system type... aarch64-apple-darwin22.4.0
checking target system type... aarch64-apple-darwin22.4.0
checking for ld... ld
checking how to print strings... printf
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... ld
checking if the linker (ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 786432
checking how to convert aarch64-apple-darwin22.4.0 file names to aarch64-apple-darwin22.4.0 format... func_convert_file_noop
checking how to convert aarch64-apple-darwin22.4.0 file names to toolchain format... func_convert_file_noop
checking for ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin22.4.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C preprocessor... gcc -E
checking C compiler vendor... clang-14-0
checking whether host executables can be run in the build... yes
checking whether #! works in shell scripts... yes
checking for flexdll sources... checking for flexlink... no
checking for flexdll.h... no
checking for a BSD-compatible install... /opt/homebrew/bin/ginstall -c
checking for cos in -lm... yes
checking for math.h... yes
checking for unistd.h... (cached) yes
checking for stdint.h... (cached) yes
checking for pthread_np.h... no
checking for dirent.h... yes
checking for sys/select.h... yes
checking for stdatomic.h... yes
checking for sys/mman.h... yes
checking for off_t... yes
checking size of int... 4
checking size of long... 8
checking size of long *... 8
checking size of short... 2
checking size of long long... 8
configure: Target is a 64 bits architecture
checking whether byte ordering is bigendian... no
checking alignment of double... 8
checking alignment of long... 8
checking alignment of long long... 8
checking whether the C compiler supports _Atomic types... yes
checking whether the C compiler supports __attribute__((aligned(n)))... yes
checking whether the C compiler supports __attribute__((optimize("tree-vectorize")))... no
checking for ld... ld
checking for rlwrap... rlwrap
configure: checking semantics of signal handlers
checking for sigaction... yes
checking for sigprocmask... yes
configure: POSIX signal handling found.
checking for expm1... yes
checking for log1p... yes
checking for hypot... yes
checking for fma... yes
checking for exp2... yes
checking for log2... yes
checking for cbrt... yes
checking for acosh... yes
checking for asinh... yes
checking for atanh... yes
checking for erf... yes
checking for erfc... yes
checking for trunc... yes
checking for round... yes
checking for copysign... yes
checking whether round works... yes
checking whether fma works... yes
checking for gcc options needed to detect all undeclared functions... none needed
checking for unistd.h... (cached) yes
checking whether getentropy is declared... no
checking for getrusage... yes
checking for times... yes
checking for secure_getenv... no
checking for __secure_getenv... no
checking for issetugid... yes
checking for clock_gettime_nsec_np... yes
checking for socket... yes
checking for socketpair... yes
checking for bind... yes
checking for listen... yes
checking for accept... yes
checking for connect... yes
checking for socklen_t... yes
checking for inet_aton... yes
checking for struct sockaddr_in6... yes
checking for getaddrinfo... yes
checking for getnameinfo... yes
checking for inet_pton... yes
checking for inet_ntop... yes
checking for rewinddir... yes
checking for lockf... yes
checking for mkfifo... yes
checking for getcwd... yes
checking whether system is declared... yes
checking for sys/types.h... (cached) yes
checking for utime.h... yes
checking for utime... yes
checking for utimes... yes
checking for fchmod... yes
checking for fchown... yes
checking for truncate... yes
checking for ftruncate... yes
checking for select... yes
checking for fd_set... yes
checking for nanosleep... yes
checking for symlink... yes
checking for readlink... yes
checking for lstat... yes
checking for realpath... yes
checking for waitpid... yes
checking for wait4... yes
checking for getgroups... yes
checking for setgroups... yes
checking for initgroups... yes
checking for termios.h... yes
checking for tcgetattr... yes
checking for tcsetattr... yes
checking for tcsendbreak... yes
checking for tcflush... yes
checking for tcflow... yes
checking for setitimer... yes
checking for gethostname... yes
checking for sys/utsname.h... yes
checking for uname... yes
checking for gettimeofday... yes
checking for mktime... yes
checking for setsid... yes
checking for putenv... yes
checking for setenv... yes
checking for unsetenv... yes
checking for locale.h... yes
checking for newlocale... yes
checking for freelocale... yes
checking for uselocale... yes
checking for xlocale.h... yes
checking for newlocale... (cached) yes
checking for freelocale... (cached) yes
checking for uselocale... (cached) yes
checking for strtod_l... yes
checking for dlopen... yes
configure: Dynamic loading of shared libraries is supported.
checking for sys/mman.h... (cached) yes
checking for mmap... yes
checking for munmap... yes
checking for pwrite... yes
checking whether the C compiler supports -fdebug-prefix-map... yes
checking for struct stat.st_atim.tv_nsec... no
checking for struct stat.st_atimespec.tv_nsec... yes
configure: stat supports nanosecond precision
checking how many arguments gethostbyname_r() takes... can't tell
checking how many arguments gethostbyaddr_r() takes... can't tell
checking for mkstemp... yes
checking for nice... yes
checking for dup3... no
checking for pipe2... no
checking for accept4... no
checking for getauxval... no
checking for sys/shm.h... yes
checking for shmat... yes
checking for execvpe... no
checking for spawn.h... yes
checking for posix_spawn... yes
checking for posix_spawnp... yes
checking for ffs... yes
checking for _BitScanForward... no
checking arguments for pthread_getaffinity_np... pthread_getaffinity_np not found
checking for pkg-config... /opt/homebrew/bin/pkg-config
configure: compressed marshaling supported
configure: ocamldebug supported
checking whether gcc is Clang... yes
checking whether pthreads work with "-pthread" and "-lpthread"... yes
checking whether Clang needs flag to prevent "argument unused" warning when linking with -pthread... no
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking whether more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for sigwait... yes
configure: the threads library is supported
checking whether the assembler supports --debug-prefix-map... no
checking whether the assembler supports CFI directives... yes
configure: not using frame pointers
checking whether mmap supports huge pages... no
checking for patdiff... no
checking for diff... diff
checking whether diff supports --strip-trailing-cr... yes
checking whether diff supports -u... yes
checking whether mmap supports MAP_STACK... no
configure: No support for function sections on aarch64-apple-darwin22.4.0.
configure: Disabling function sections.
checking for backtrace in -lexecinfo... no
configure: creating ./config.status
config.status: creating Makefile.build_config
config.status: creating Makefile.config
config.status: creating stdlib/sys.ml
config.status: creating manual/src/version.tex
config.status: creating manual/src/html_processing/src/common.ml
config.status: creating ocamltest/ocamltest_config.ml
config.status: creating utils/config.generated.ml
config.status: creating compilerlibs/META
config.status: creating otherlibs/dynlink/META
config.status: creating otherlibs/runtime_events/META
config.status: creating stdlib/META
config.status: creating otherlibs/unix/META
config.status: creating otherlibs/str/META
config.status: creating otherlibs/systhreads/META
config.status: creating ocamldoc/META
config.status: creating runtime/caml/m.h
config.status: creating runtime/caml/s.h
config.status: creating runtime/caml/version.h
config.status: linking otherlibs/unix/unix_unix.ml to otherlibs/unix/unix.ml
config.status: executing libtool commands

and sh -x ./configure

Looking at that log (I am not myself familiar with the build system) the way the path to the lib is obtained is by calling: /opt/homebrew/bin/pkg-config --libs libzstd and it returns -L/opt/homebrew/Cellar/zstd/1.5.5/lib -lzstd.

I think the reliance on pkg-config (and not on some specific brew --cellar command) makes the script compatible with non-brew systems. But pkg-config reads from the .pc file written by brew, which contains a fragile link to a versioned folder instead of a link to /opt/homebrew/lib.

Side question: does brew install ocaml fit your use case or do you need to build it from source?

No the dependency to zstd is new to the yet-unreleased 5.1 version of the compiler.

The .pc file for libzstd could very well point to /opt/homebrew/lib as the location of -lzstd, since there's a symbolic link there to the actual location.

Actually, yeah that's right.

It looks like this would be a solution to our issue. Is there anything we can do to help that change ?

[carlocab]

Homebrew/homebrew-core#138187 should fix this for zstd. But, in general, recording configure-time generated paths doesn't work great for things that Homebrew installs.

[voodoos]

Great, thanks for the quick fix!

[voodoos]

The fix war merged and after reinstalling zstd I get the generic output when calling pkg-config, so we can probably close this issue now 🙂

Thanks again @osalbahr and @carlocab for your efficiency !

[kit-ty-kate]

Could this be reopened per https://discuss.ocaml.org/t/unable-to-install-some-packages-with-opam-on-macos/14434/7 ?

[Octachron]

I am reopening this issue because it still trips up homebrew users (I am seen one more report this week for instance).

[carlocab]

If there are instances that are not fixed by brew update && brew reinstall zstd, let me know.

[LangInteger]

@carlocab

I am in this issue. Please see the following console output:

liulang@lang parser % brew update && brew reinstall zstd
==> Updating Homebrew...
Already up-to-date.
==> Downloading https://ghcr.io/v2/homebrew/core/zstd/manifests/1.5.6
Already downloaded: /Users/liulang/Library/Caches/Homebrew/downloads/29403e0df5404d8aeca0e750ac135ec9ef44fc5eeb6df69170ed602acabf0ffb--zstd-1.5.6.bottle_manifest.json
==> Fetching zstd
==> Downloading https://ghcr.io/v2/homebrew/core/zstd/blobs/sha256:2028141683f55bffcd0693b9e49eef1e3dabc1e184214cacb173ca9bd54dabc0
Already downloaded: /Users/liulang/Library/Caches/Homebrew/downloads/af6fd07ad9cd935975dd761af51c9baba7eb782dda7630d4abd399fa98e53578--zstd--1.5.6.arm64_sonoma.bottle.tar.gz
==> Reinstalling zstd 
==> Pouring zstd--1.5.6.arm64_sonoma.bottle.tar.gz
🍺  /opt/homebrew/Cellar/zstd/1.5.6: 32 files, 2.2MB
==> Running `brew cleanup zstd`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
liulang@lang parser % 
liulang@lang parser % 
liulang@lang parser % 
liulang@lang parser % 
liulang@lang parser % dune build 
File "dune", line 5, characters 17-28:
5 |  (names analyzer parser_main input_main)
                     ^^^^^^^^^^^
ld: warning: directory not found for option '-L/opt/homebrew/Cellar/zstd/1.5.5/lib'
ld: library not found for -lzstd
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)
File "dune", line 5, characters 29-39:

The opam and OCaml info:

liulang@lang parser % opam --version
2.1.5
liulang@lang parser % 
liulang@lang parser % 
liulang@lang parser % ocaml --version
The OCaml toplevel, version 5.1.0

[carlocab]

How did you install ocaml/opam/dune? What project are you trying to build that gives you this error?

[LangInteger]

@carlocab

Hi, I used brew to install opam, and then use opam to install dune/ocaml related things as described at Installing OCaml.

The project I am trying to build with dune does not matter. Every ocaml project that I build now give me the error.

[dbuenzli]

It seems you don't have the latest opam and ocaml from brew. Perhaps you are stuck in a configuration minima that still exhibits the behaviour (?). At that point I would brew upgrade Make sure you get ocaml 5.2.0 and opam 2.2.1, do a rm -rf ~/.opam and start over with opam init.

[LangInteger]

@dbuenzli @carlocab
Thx! After using brew upgrade to upgrade opam to 2.2.1, rm -rf ~/.opam and start over with opam init, now my local env works without complaining the zstd problem.