static binaries failed on macOS from version 15~18

[shenxianpeng]

I tested the new pre-release master-a13caef5, the static binaries on macOS still has the problem from version 15~18.

dyld[3039]: Library not loaded: /usr/local/opt/zstd/lib/libzstd.1.dylib
  Referenced from: <54D844F3-D2D4-323E-9B2B-6EC303AF4B79> /Users/runner/work/clang-tools-static-binaries/clang-tools-static-binaries/clang-format-16_macosx-amd64
  Reason: tried: '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file, not in dyld cache)
/Users/runner/work/_temp/31e31703-4e2c-489d-84ab-c0645130b52c.sh: line 5:  3039 Abort trap: 6           ./clang-format-16_macosx-amd64 --version

More details please see the test job https://github.com/cpp-linter/clang-tools-static-binaries/actions/runs/9353719255/job/25744898037#step:3:40

Originally posted by @shenxianpeng in https://github.com/cpp-linter/clang-tools-static-binaries/issues/16#issuecomment-2145636865

[2bndy5]

I was looking at what recently changed in the MacOS runners, and I found that XCode v15.4.0 was updated to a stable release (see this PR). Maybe there was some other change in the std lib that is included with XCode v15.4.0?

EDIT: See XCode changelog here

[shenxianpeng]

I don't understand why only versions 15~18 were affected not all versions on MacOS. maybe it is also related to the llvm-project source code from version 15 to 18?

[2bndy5]

I'm not sure why the binaries are linked to a zstd library. Recently, the macos-14 image had bumped zstd from v1.5.5 to v1.5.6.

[shenxianpeng]

maybe I can try to create a link for zstd lib

sudo ln -s /opt/homebrew/Cellar/zstd/1.5.6/lib/libzstd.1.5.6.dylib /usr/lib/libzstd.1.dylib

# or just

sudo ln -s /opt/homebrew/Cellar/zstd/1.*/lib/libzstd.1.*.dylib /usr/lib/libzstd.1.dylib

[2bndy5]

The wildcard might be better, but you'd have to recreate that symbol link on every Mac machine that uses the static binary.

[2bndy5]

I skimmed the zstd lib's manual, and I found

Advanced experimental functions can be accessed using #define ZSTD_STATIC_LINKING_ONLY before including zstd.h.

Advanced experimental APIs should never be used with a dynamically-linked library. They are not "stable"; their definitions or signatures may change in the future. Only static linking is allowed.

I haven't checked the workflow to see if mac builds use -DZSTD_STATIC_LINKING_ONLY=1

[shenxianpeng]

use -DZSTD_STATIC_LINKING_ONLY=1 works 👍

[shenxianpeng]

From the test the following issue still occurs when running clang-format --version command, so I reopened this ticket.

== Output clang-format version
dyld[3452]: Library not loaded: /usr/local/opt/zstd/lib/libzstd.1.dylib
  Referenced from: <182FA4B3-05C4-32D1-B379-F93C1FA28DC0> /Users/runner/work/clang-tools-static-binaries/clang-tools-static-binaries/clang-format-15_macosx-amd64
  Reason: tried: '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file, not in dyld cache)
/Users/runner/work/_temp/d23ec023-0cb7-410c-9fcd-8c17b55eae17.sh: line 5:  3452 Abort trap: 6           ./clang-format-15_macosx-amd64 --version
Error: Process completed with exit code 134.

It's my bad, use -DZSTD_STATIC_LINKING_ONLY=1 does not make the build fail, my brain just assumed the problem was solved 😕

[2bndy5]

Well, we haven't released the new builds yet. I don't think the current test CI uses pre-releases. So I think that error is just from running the last (erroneous) release's builds, specifically the release that github thinks is the "latest": https://github.com/cpp-linter/clang-tools-static-binaries/releases/latest.

https://github.com/cpp-linter/clang-tools-static-binaries/blob/2e4f8c1fcb85fb322cdf24789f98de42f857a014/.github/workflows/test.yml#L33

gh release download -h

``` Download assets from a GitHub release. Without an explicit tag name argument, assets are downloaded from the latest release in the project. In this case, `--pattern` or `--archive` is required. USAGE gh release download [] [flags] FLAGS -A, --archive format Download the source code archive in the specified format (zip or tar.gz) --clobber Overwrite existing files of the same name -D, --dir directory The directory to download files into (default ".") -O, --output file The file to write a single asset to (use "-" to write to standard output) -p, --pattern stringArray Download only assets that match a glob pattern --skip-existing Skip downloading when files of the same name exist INHERITED FLAGS --help Show help for command -R, --repo [HOST/]OWNER/REPO Select another repository using the [HOST/]OWNER/REPO format EXAMPLES # download all assets from a specific release $ gh release download v1.2.3 # download only Debian packages for the latest release $ gh release download --pattern '*.deb' # specify multiple file patterns $ gh release download -p '*.deb' -p '*.rpm' # download the archive of the source code for a release $ gh release download v1.2.3 --archive=zip LEARN MORE Use `gh --help` for more information about a command. Read the manual at https://cli.github.com/manual ```

We could add a tag named latest and keep moving the tag when we publish a release (or pre-release). That way we can have the test CI use gh release download latest <tool-name>.

---

For proof that the seg fault was fixed see this CI run for cpp-linter/clang-tools-pip#99

[shenxianpeng]

Thank you for pointing out @2bndy5

I have added tag as input for the test CI, and tested the pre-release build successfully.

https://github.com/cpp-linter/clang-tools-static-binaries/blob/1bcb98a6219f62b9db46d863b5cc21edeaa79452/.github/workflows/test.yml#L39